Nov 03 22:19:23.719023 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 22:19:23.719035 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 22:19:23.719063 ip-10-0-135-87 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 22:19:23.719357 ip-10-0-135-87 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 22:19:33.893095 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 22:19:33.893110 ip-10-0-135-87 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 937c0b06d9f04dfe9102feb7054c86dc -- Nov 03 22:21:23.612313 ip-10-0-135-87 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:21:24.018232 ip-10-0-135-87 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:24.018232 ip-10-0-135-87 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:21:24.018232 ip-10-0-135-87 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:24.018232 ip-10-0-135-87 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:24.023275 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023198 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.023275 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023274 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.023275 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023278 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023281 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023284 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023290 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023292 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023298 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023304 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023307 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023309 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023312 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023315 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023317 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023320 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023322 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023326 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023328 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.023365 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023331 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023334 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023337 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023339 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023342 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023345 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023347 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023350 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023353 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023355 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023358 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023360 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023365 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023368 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023371 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023374 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023376 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023380 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.023834 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023382 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023386 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023388 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023391 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023393 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023398 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023401 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023403 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023405 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023408 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023413 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023415 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023417 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023421 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023425 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023429 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023432 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.024403 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023435 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023438 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023441 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023443 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023446 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023449 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023452 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023454 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023457 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023459 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023461 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023464 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023466 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023469 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023472 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023474 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023478 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023482 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023485 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023487 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.024876 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023490 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023492 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023495 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023497 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023500 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023502 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023505 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023507 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023510 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023514 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023516 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023519 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023522 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023524 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023527 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023529 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023532 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023534 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023537 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023539 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.025371 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023543 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023546 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023548 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023551 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023553 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023556 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.023558 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024374 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024381 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024384 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024387 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024390 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024393 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024395 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024398 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024401 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024403 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024406 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024409 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.025832 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024411 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024414 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024418 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024420 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024423 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024426 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024428 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024431 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024433 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024436 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024438 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024441 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024443 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024445 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024448 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024450 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024453 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024456 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024458 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024460 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.026304 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024475 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024479 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024482 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024485 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024488 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024490 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024493 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024495 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024498 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024500 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024503 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024505 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024509 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024511 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024514 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024516 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024519 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024522 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.026779 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024524 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024527 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024529 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024532 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024534 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024536 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024539 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024541 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024544 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024547 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024550 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024552 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024555 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024558 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024561 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024564 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024567 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024569 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024572 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024575 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.027253 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024577 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024579 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024582 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024584 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024587 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024590 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024593 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024595 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024598 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024600 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024603 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024605 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024608 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024610 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024613 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024615 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024617 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024621 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024623 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.027749 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024626 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024630 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024637 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024639 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024642 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024644 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024647 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024650 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024653 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024656 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024659 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024662 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024666 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024669 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.024671 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025165 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025170 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025173 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025175 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.028218 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025178 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025181 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025185 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025189 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025192 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025195 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025198 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025200 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025203 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025205 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025208 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025210 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025213 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025215 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025218 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025220 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025223 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025225 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025228 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.028690 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025230 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025236 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025241 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025244 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025246 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025249 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025251 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025254 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025256 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025259 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025261 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025264 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025266 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025269 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025271 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025273 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025276 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025278 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.029152 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025281 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025284 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025286 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025288 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025292 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025296 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025299 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025302 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025304 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025307 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025310 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025313 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025316 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025318 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025324 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025327 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025329 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025332 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025334 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.029613 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025336 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025339 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025341 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025344 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025347 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025349 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025351 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025354 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025357 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025359 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025364 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025367 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025369 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025372 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025374 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025377 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025379 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025382 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025384 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.030071 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025387 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025390 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025392 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025395 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025397 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025400 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025402 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025405 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025408 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025411 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025413 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025416 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025418 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025421 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025423 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025426 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025428 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025430 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025433 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025435 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.030527 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025440 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025443 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.025446 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026223 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026233 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026239 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026244 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026248 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026251 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026258 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026262 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026268 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026271 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026274 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026277 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026281 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026284 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026286 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026290 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026292 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026296 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026301 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026303 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:21:24.031000 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026307 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026310 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026313 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026317 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026320 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026323 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026326 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026329 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026332 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026335 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026338 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026341 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026345 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026348 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026351 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026354 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026357 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026359 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026363 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026366 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026370 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026374 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026376 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026380 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:21:24.031555 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026383 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026386 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026389 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026392 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026396 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026398 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026402 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026405 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026408 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026411 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026415 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026418 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026421 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026424 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026427 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026430 2578 flags.go:64] FLAG: --help="false" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026433 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026436 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026439 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026442 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026446 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026449 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026452 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:21:24.032162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026454 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026457 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026460 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026462 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026465 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026469 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026472 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026475 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026478 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026481 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026484 2578 flags.go:64] FLAG: --lock-file="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026487 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026490 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026493 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026498 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026501 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026505 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026507 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026510 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026514 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026517 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026519 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026524 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026527 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026531 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 22:21:24.032737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026534 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026536 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026539 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026542 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026545 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026548 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026551 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026558 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026561 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026564 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026567 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026570 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026574 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026577 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026580 2578 flags.go:64] FLAG: --port="10250" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026583 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026585 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0aabb26406964e5b8" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026588 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026591 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026595 2578 flags.go:64] FLAG: --register-node="true" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026598 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026602 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026605 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026608 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026611 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026616 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:21:24.033334 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026619 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026621 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026624 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026627 2578 flags.go:64] FLAG: --runonce="false" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026630 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026633 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026636 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026639 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026642 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026645 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026648 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026651 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026654 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026656 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026659 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026662 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026665 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026668 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026672 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026677 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026680 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026683 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026687 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026690 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026692 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:21:24.034123 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026697 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026700 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026703 2578 flags.go:64] FLAG: --v="2" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026708 2578 flags.go:64] FLAG: --version="false" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026712 2578 flags.go:64] FLAG: --vmodule="" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026716 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.026719 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026859 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026863 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026866 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026869 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026871 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026874 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026876 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026879 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026881 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026884 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026888 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026891 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026894 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.035033 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026896 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026898 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026901 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026904 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026908 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026910 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026913 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026916 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026918 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026921 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026923 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026927 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026930 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026932 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026935 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026937 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026940 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026943 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026945 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026948 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.035688 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026950 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026953 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026956 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026958 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026960 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026963 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026965 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026968 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026970 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026973 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026975 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026977 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026980 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026983 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.026999 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027002 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027006 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027008 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027011 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.036187 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027014 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027016 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027019 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027022 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027025 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027028 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027030 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027033 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027035 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027038 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027040 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027042 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027046 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027048 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027051 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027053 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027056 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027058 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027061 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.036636 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027063 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027066 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027068 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027070 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027073 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027075 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027078 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027080 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027083 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027085 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027089 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027093 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027096 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027099 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027101 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027104 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027106 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027110 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027113 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027115 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.037126 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027118 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027120 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027123 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027127 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027130 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027133 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027140 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027143 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027145 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027148 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027150 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027153 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027155 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027157 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.027186 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027225 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.037591 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027229 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027232 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027235 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027237 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027244 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027249 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027253 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027256 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027259 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027262 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027265 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027268 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027272 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027275 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027278 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027281 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027283 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027286 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.037973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027289 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027291 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027294 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027296 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027299 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027302 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027305 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027310 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027312 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027315 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027317 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027320 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027322 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027325 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027327 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027330 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027332 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027335 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027338 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.038436 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027341 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027343 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027346 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027348 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027351 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027353 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027355 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027359 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027362 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027365 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027367 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027369 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027372 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027375 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027377 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027380 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027382 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027385 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027387 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.038900 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027390 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027392 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027398 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027401 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027404 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027406 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027409 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027411 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027414 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027416 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027418 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027421 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027423 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027426 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027429 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027431 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027434 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027436 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027438 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.039355 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027442 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027444 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027447 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027449 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027452 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027456 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027459 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027461 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027464 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027466 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027469 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027472 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027474 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027479 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027482 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027484 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027487 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.039820 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027489 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027491 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027495 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027499 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027503 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027505 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027508 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.027510 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.034899 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.034982 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.035753 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.035760 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035828 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035834 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035837 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035840 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035843 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035846 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035849 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.040286 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035852 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035855 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035857 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035860 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035862 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035867 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035870 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035873 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035875 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035878 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035881 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035883 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035886 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035888 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035892 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035895 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035898 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035900 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.040726 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035903 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035906 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035908 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035912 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035915 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035918 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035920 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035923 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035925 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035928 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035931 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035933 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035935 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035938 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035941 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035943 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035946 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035948 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035951 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035953 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.041217 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035956 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035958 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035961 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035963 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035965 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035968 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035971 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035973 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.035977 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036008 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036012 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036015 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036018 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036021 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036025 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036027 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036030 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036033 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036036 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.041694 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036038 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036041 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036043 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036046 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036048 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036051 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036053 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036056 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036059 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036062 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036064 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036066 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036069 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036071 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036075 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036078 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036082 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036086 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036089 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036091 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.042183 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036094 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036097 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036099 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036102 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036104 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036107 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036109 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036112 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036114 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036117 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036119 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036122 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036125 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036127 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036130 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036133 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036135 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036140 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036143 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.042638 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036145 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.036174 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036313 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036318 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036321 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036323 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036326 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036330 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036333 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036335 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036338 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036340 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036343 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036345 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036348 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.043135 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036350 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036353 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036355 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036358 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036361 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036366 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036368 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036370 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036373 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036375 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036379 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036383 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036385 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036388 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036390 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036393 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036396 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036398 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.043559 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036401 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036403 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036405 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036408 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036410 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036413 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036416 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036418 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036421 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036423 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036426 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036429 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036431 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036433 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036436 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036439 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036441 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036444 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036446 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036448 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.044026 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036451 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036453 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036455 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036458 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036460 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036463 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036465 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036468 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036470 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036473 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036475 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036477 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036480 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036482 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036485 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036487 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036490 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036492 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036495 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.044532 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036498 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036500 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036503 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036506 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036510 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036513 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036516 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036519 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036522 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036524 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036527 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036530 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036532 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036535 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036539 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036542 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036544 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036547 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.044973 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036549 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036552 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036554 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036557 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036559 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036562 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036564 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036566 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036569 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036571 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036573 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036576 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036578 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036581 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036584 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036586 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.036610 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:24.045424 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036647 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036651 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036654 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036656 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036659 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036661 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036666 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036669 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036671 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036674 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036676 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036679 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036681 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036684 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036686 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036688 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036691 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036693 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036696 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036698 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:24.045857 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036701 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036703 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036708 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036711 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036713 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036715 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036718 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036721 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036724 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036729 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036732 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036735 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036737 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036739 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036742 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036744 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036747 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036751 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:24.046336 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036754 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036756 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036759 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036761 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036763 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036766 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036768 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036771 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036773 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036776 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036781 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036784 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036786 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036788 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036791 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036793 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036796 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036798 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036801 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036803 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:24.046767 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036806 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036808 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036811 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036813 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036816 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036819 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036821 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036823 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036826 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036828 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036831 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036834 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036837 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036839 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036841 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036844 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036847 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036849 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036852 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:24.047245 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036854 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036857 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036859 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036863 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036865 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036868 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036872 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036875 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036877 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036880 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036883 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036885 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036888 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036890 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036893 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036895 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036897 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036900 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036903 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036905 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:24.047684 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036908 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036910 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036915 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036919 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036922 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036925 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:24.036927 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.036978 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.043179 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.044045 2578 server.go:1014] "Starting client certificate rotation" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.044140 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.044175 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.045764 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:24.048273 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.046964 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:24.054577 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.054559 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:21:24.057497 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.057482 2578 log.go:25] "Validated CRI v1 image API" Nov 03 22:21:24.058323 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.058296 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:21:24.063857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.063836 2578 fs.go:135] Filesystem UUIDs: map[48a23b4c-f1cf-4b70-8724-37ccdfa68e72:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d3c1b729-ee41-4203-bbbf-b276f817e818:/dev/nvme0n1p4] Nov 03 22:21:24.063907 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.063857 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:21:24.069455 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.069335 2578 manager.go:217] Machine: {Timestamp:2026-11-03 22:21:24.068169891 +0000 UTC m=+0.342850217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123091 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23f5dc7060e2436d16b1e1bf75ba5b SystemUUID:ec23f5dc-7060-e243-6d16-b1e1bf75ba5b BootID:937c0b06-d9f0-4dfe-9102-feb7054c86dc Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:e8:6c:f8:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:e8:6c:f8:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:d4:78:b3:75:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:21:24.069455 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.069451 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:21:24.069569 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.069537 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:21:24.071588 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071556 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:21:24.071761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071592 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-87.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:21:24.071837 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071766 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:21:24.071837 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071780 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:21:24.071837 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071817 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:24.071837 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.071835 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:24.072531 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.072517 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:21:24.075891 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.075876 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:21:24.076065 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076054 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:21:24.076669 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076658 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:21:24.076721 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076677 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:21:24.076721 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076693 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:21:24.076721 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076708 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:21:24.076721 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.076720 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:21:24.077174 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.077149 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:24.080948 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.080928 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:21:24.084723 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.084711 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:24.085462 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085450 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:21:24.085493 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085481 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:21:24.085653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085642 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:21:24.085681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085660 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:21:24.085681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085666 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:21:24.085681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085672 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:21:24.085681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085677 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:21:24.085681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.085682 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:21:24.086144 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086135 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:21:24.086173 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086146 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:21:24.086173 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086154 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:21:24.086173 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086160 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:21:24.086173 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086168 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:21:24.086373 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086177 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:21:24.086702 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086691 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:21:24.086741 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.086705 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:21:24.090434 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.090421 2578 server.go:1263] "Started kubelet" Nov 03 22:21:24.090796 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.090492 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:21:24.090796 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.090536 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:21:24.090796 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.090608 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:21:24.091281 ip-10-0-135-87 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:21:24.091656 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.091487 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:21:24.092311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.092258 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:21:24.097515 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.097489 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:24.097515 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.097494 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:21:24.097791 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.097777 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:21:24.097904 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.097867 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:24.098397 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098285 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:21:24.098495 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098482 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:21:24.098550 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098498 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:21:24.098681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098664 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:21:24.098681 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098678 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:21:24.098917 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098904 2578 factory.go:153] Registering CRI-O factory Nov 03 22:21:24.098957 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098919 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 22:21:24.098957 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098956 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:21:24.099046 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098962 2578 factory.go:55] Registering systemd factory Nov 03 22:21:24.099046 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098968 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:21:24.099046 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.098980 2578 factory.go:103] Registering Raw factory Nov 03 22:21:24.099046 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.099014 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 22:21:24.100037 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.100021 2578 manager.go:319] Starting recovery of all containers Nov 03 22:21:24.110936 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.110912 2578 manager.go:324] Recovery completed Nov 03 22:21:24.115857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.115839 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.117586 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.117570 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.117654 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.117600 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.117654 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.117611 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.118095 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118081 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:21:24.118095 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118094 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:21:24.118199 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118099 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.118199 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118114 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:21:24.118737 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118722 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.118790 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118746 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.118790 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.118755 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.120352 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.120339 2578 policy_none.go:50] "Start" Nov 03 22:21:24.120415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.120359 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:21:24.120415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.120373 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:21:24.122380 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.122366 2578 policy_none.go:44] "Start" Nov 03 22:21:24.149386 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.149355 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:21:24.154332 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154315 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 22:21:24.154405 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.154348 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:21:24.154405 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154358 2578 server.go:87] "Starting device plugin registration server" Nov 03 22:21:24.154647 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154633 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:21:24.154707 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154648 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:21:24.154746 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154740 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154808 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.154815 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.155729 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.155778 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.170159 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.171457 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.171476 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.171493 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:21:24.180979 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.171523 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:21:24.239895 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.239737 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-87.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:21:24.239895 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.239746 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-87.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:21:24.240629 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.239735 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-87.us-west-1.compute.internal.18e4a91ac80632ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-87.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-87.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:24.090393259 +0000 UTC m=+0.365073583,LastTimestamp:2026-11-03 22:21:24.090393259 +0000 UTC m=+0.365073583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-87.us-west-1.compute.internal,}" Nov 03 22:21:24.256189 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.256167 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.257007 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.256977 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.257082 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.257025 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.257082 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.257036 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.257082 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.257060 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.272621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.272562 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal"] Nov 03 22:21:24.272702 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.272629 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.273436 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.273422 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.273505 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.273448 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.273505 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.273459 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.275661 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.275649 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.275798 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.275785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.275853 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.275812 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.276434 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276418 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.276527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276448 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.276527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276462 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.276527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276418 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.276527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276516 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.276527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.276526 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.278537 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.278520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.278631 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.278552 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.279264 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.279240 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.279264 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.279268 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.279402 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.279277 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.298471 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.298442 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.298803 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.298788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.298859 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.298818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.298859 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.298845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.304206 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.304189 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.324790 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.324775 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:24.324868 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.324836 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkmp5" Nov 03 22:21:24.329861 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.329777 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-87.us-west-1.compute.internal.18e4a91ac9a5209e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-87.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-87.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-87.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:24.117586078 +0000 UTC m=+0.392266403,LastTimestamp:2026-11-03 22:21:24.117586078 +0000 UTC m=+0.392266403,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-87.us-west-1.compute.internal,}" Nov 03 22:21:24.330173 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.330148 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-87.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:21:24.330216 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.330144 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 22:21:24.330216 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.330151 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:21:24.399454 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.399454 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.399624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.399624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.399624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.399624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.399509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.501235 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.501206 2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-87.us-west-1.compute.internal\" not found" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.503773 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.503756 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:24.553148 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.553094 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-87.us-west-1.compute.internal\" not found" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.601110 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.601074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.607323 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.606843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.701938 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.701911 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:24.702940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.702924 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:24.703014 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.702957 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:24.703014 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.702973 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:24.703014 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.703010 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.761438 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:24.761414 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:24.761527 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.761445 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-87.us-west-1.compute.internal\": node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:24.888753 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.888725 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:24.988867 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:24.988829 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:25.044022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.043975 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:21:25.044429 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.044202 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:21:25.044429 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:25.044183 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6c0d62e96a584816a3ee7bc88cce2ee-6000754553a9ad3e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-87.us-west-1.compute.internal.18e4a91ac9a59221\": read tcp 10.0.135.87:39134->3.15.56.150:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-87.us-west-1.compute.internal.18e4a91ac9a59221 default 4771 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-87.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-87.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-87.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:24 +0000 UTC,LastTimestamp:2026-11-03 22:21:24.257039748 +0000 UTC m=+0.531720073,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-87.us-west-1.compute.internal,}" Nov 03 22:21:25.089507 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:25.089469 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:25.190095 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:25.190007 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:25.290385 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:25.290351 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:25.324058 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.324031 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:25.324899 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.324880 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:25.325029 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.324917 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:25.326687 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.325245 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:25.329316 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.329292 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:25 +0000 UTC" deadline="2028-08-17 04:32:34.402213724 +0000 UTC" Nov 03 22:21:25.329316 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.329315 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15654h11m9.072900729s" Nov 03 22:21:25.391204 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:25.391181 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:21:25.457177 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.457113 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:25.482463 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.482440 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:25.498569 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.498547 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:25.559001 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.558974 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:25.559098 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.559086 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:21:25.622623 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.622604 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:25.772128 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:25.772089 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6400a77c4e4822387f7b4ea34fe62bb6.slice/crio-5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f WatchSource:0}: Error finding container 5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f: Status 404 returned error can't find the container with id 5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f Nov 03 22:21:25.772531 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:25.772513 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1edcc8793fd6be957384c61df114fef.slice/crio-429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1 WatchSource:0}: Error finding container 429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1: Status 404 returned error can't find the container with id 429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1 Nov 03 22:21:25.776953 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.776939 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:21:25.971928 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:25.971890 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:26.079203 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.079113 2578 apiserver.go:52] "Watching apiserver" Nov 03 22:21:26.098783 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.098753 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:26.176248 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.176194 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerStarted","Data":"429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1"} Nov 03 22:21:26.177389 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.177366 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"6400a77c4e4822387f7b4ea34fe62bb6","Type":"ContainerStarted","Data":"5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f"} Nov 03 22:21:26.249201 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.249178 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:21:26.249493 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.249476 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j","openshift-multus/multus-additional-cni-plugins-tzg8f","openshift-dns/node-resolver-dw62d","openshift-image-registry/node-ca-6hzqr","openshift-multus/network-metrics-daemon-m8g4v","openshift-ovn-kubernetes/ovnkube-node-cdcgs","kube-system/global-pull-secret-syncer-dchqn","kube-system/konnectivity-agent-mlhk9","kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-wmt2m","openshift-network-operator/iptables-alerter-4pjvv","openshift-cluster-node-tuning-operator/tuned-dcbzg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal","openshift-multus/multus-h85gn"] Nov 03 22:21:26.253524 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.253503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.253626 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.253591 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:26.255683 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.255666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.257941 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.257923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.260168 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.260146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.262504 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.262483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.262583 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.262562 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:26.264939 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.264921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.267416 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.267398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.269756 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.269743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.272373 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.272353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.275437 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.275414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:26.275532 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.275490 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:26.277947 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.277932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.281051 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.281032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.299100 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.299059 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:21:26.311042 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmbh\" (UniqueName: \"kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311265 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.311265 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.311265 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311265 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.311415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.311415 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.311600 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxqb\" (UniqueName: \"kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlc8g\" (UniqueName: \"kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.311812 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311795 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlkq\" (UniqueName: \"kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.311987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdjq\" (UniqueName: \"kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdl4b\" (UniqueName: \"kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312208 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwf4\" (UniqueName: \"kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.312695 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64c4\" (UniqueName: \"kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.312965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zq6l\" (UniqueName: \"kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313675 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313758 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313777 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.313994 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.313802 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-px8pz\"" Nov 03 22:21:26.363061 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363009 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.363061 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363061 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q82l2\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363059 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363116 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dbbtb\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363126 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2wh6\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363149 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglhz\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363209 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.363280 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363267 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:21:26.363562 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363348 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.363562 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363360 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:21:26.363562 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363369 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dx5sz\"" Nov 03 22:21:26.363562 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363394 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.363562 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363478 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.363757 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363670 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:21:26.363757 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363676 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:21:26.364019 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363948 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.364019 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363958 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.364019 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363954 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:21:26.364019 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363972 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:21:26.364019 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.363988 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364071 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364092 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364098 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364143 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364149 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.364311 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.364184 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7mwkp" Nov 03 22:21:26.366357 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.366344 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.368883 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.368865 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:21:26.414741 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.414858 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.414858 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.414858 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.414858 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414870 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.414954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64c4\" (UniqueName: \"kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.415048 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:26.415077 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415047 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415141 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.415155 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:26.915107903 +0000 UTC m=+3.189788234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.415582 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415436 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415479 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq6l\" (UniqueName: \"kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.416471 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415733 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmbh\" (UniqueName: \"kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417139 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.415987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416219 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.417940 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxqb\" (UniqueName: \"kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlc8g\" (UniqueName: \"kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.416506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417082 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlkq\" (UniqueName: \"kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdjq\" (UniqueName: \"kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdl4b\" (UniqueName: \"kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.419952 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.417444 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.417493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:26.917477395 +0000 UTC m=+3.192157723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwf4\" (UniqueName: \"kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.420852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.417979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418406 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418431 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418446 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2scz\"" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418565 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mx976\"" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.418798 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhswt\"" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.420912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.420952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.421427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.421114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.426478 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.426456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.428431 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.428414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.481935 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.481902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64c4\" (UniqueName: \"kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.482049 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.481974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zq6l\" (UniqueName: \"kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.482614 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.482592 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmbh\" (UniqueName: \"kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.482684 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.482668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlc8g\" (UniqueName: \"kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.483479 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.483462 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:26.483571 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.483484 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:26.483571 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.483498 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:26.483571 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.483559 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:26.983540796 +0000 UTC m=+3.258221115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:26.485973 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.485947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdjq\" (UniqueName: \"kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.486190 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.486168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxqb\" (UniqueName: \"kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.486247 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.486224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlkq\" (UniqueName: \"kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.486247 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.486235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdl4b\" (UniqueName: \"kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.486459 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.486444 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwf4\" (UniqueName: \"kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.569527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.569495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:21:26.572717 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.572698 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:26.575178 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.575164 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dw62d" Nov 03 22:21:26.580363 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.580338 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ac9b7f_736c_405d_babf_089ef215fb8a.slice/crio-c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2 WatchSource:0}: Error finding container c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2: Status 404 returned error can't find the container with id c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2 Nov 03 22:21:26.581395 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.581379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:21:26.584314 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.584290 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8595e2d4_2ee4_41e4_8844_eae4e7ecfdb1.slice/crio-a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a WatchSource:0}: Error finding container a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a: Status 404 returned error can't find the container with id a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a Nov 03 22:21:26.590501 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.590469 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450acb37_2b14_4264_b470_ce0a63835e8b.slice/crio-ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885 WatchSource:0}: Error finding container ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885: Status 404 returned error can't find the container with id ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885 Nov 03 22:21:26.600802 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.600783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h85gn" Nov 03 22:21:26.602138 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.602118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:26.606570 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.606552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:26.611009 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.610975 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a5d53c_512b_48d9_be21_a6f82b3f3761.slice/crio-e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78 WatchSource:0}: Error finding container e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78: Status 404 returned error can't find the container with id e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78 Nov 03 22:21:26.616944 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.616921 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8192fe_9bfe_4b5e_ab36_977ddf31533e.slice/crio-8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63 WatchSource:0}: Error finding container 8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63: Status 404 returned error can't find the container with id 8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63 Nov 03 22:21:26.623542 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.623523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:21:26.628779 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.628762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:21:26.630664 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.630646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:21:26.635652 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.635628 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a01977_b5b7_4177_bfd4_e5d2265b442e.slice/crio-6915ac4ae6df067c9e4c0425f2eefa5c1c6edb6d281678d7fb085df203b807e3 WatchSource:0}: Error finding container 6915ac4ae6df067c9e4c0425f2eefa5c1c6edb6d281678d7fb085df203b807e3: Status 404 returned error can't find the container with id 6915ac4ae6df067c9e4c0425f2eefa5c1c6edb6d281678d7fb085df203b807e3 Nov 03 22:21:26.639905 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:26.639883 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597ba6ab_ea33_4347_871a_f4edc6188d05.slice/crio-802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c WatchSource:0}: Error finding container 802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c: Status 404 returned error can't find the container with id 802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c Nov 03 22:21:26.920792 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.920703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:26.920792 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:26.920786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:26.921119 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.920846 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:26.921119 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.920893 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:26.921119 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.920922 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:27.920900646 +0000 UTC m=+4.195580962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:26.921119 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:26.920947 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:27.920931954 +0000 UTC m=+4.195612281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:27.021669 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.021630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:27.021862 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.021822 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:27.021862 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.021844 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:27.021974 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.021868 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:27.021974 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.021922 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:28.021909029 +0000 UTC m=+4.296589356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:27.181585 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.181489 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerStarted","Data":"9ebf34f0bf961d7ea1bdb48a89f7b2171ce0dc1b5b9fc65aa046854179798295"} Nov 03 22:21:27.183162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.183129 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" event={"ID":"597ba6ab-ea33-4347-871a-f4edc6188d05","Type":"ContainerStarted","Data":"802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c"} Nov 03 22:21:27.185704 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.184608 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" event={"ID":"d6a01977-b5b7-4177-bfd4-e5d2265b442e","Type":"ContainerStarted","Data":"6915ac4ae6df067c9e4c0425f2eefa5c1c6edb6d281678d7fb085df203b807e3"} Nov 03 22:21:27.186253 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.186229 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63"} Nov 03 22:21:27.187823 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.187796 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mlhk9" event={"ID":"661b34e8-d977-46ae-a3c6-6f09a1182f8d","Type":"ContainerStarted","Data":"30c152cb9d7ddbd2848b5a4e0266d4e06c45d8503af62670f94a7718a41cefd0"} Nov 03 22:21:27.189190 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.189142 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h85gn" event={"ID":"87a5d53c-512b-48d9-be21-a6f82b3f3761","Type":"ContainerStarted","Data":"e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78"} Nov 03 22:21:27.190596 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.190566 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hzqr" event={"ID":"450acb37-2b14-4264-b470-ce0a63835e8b","Type":"ContainerStarted","Data":"ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885"} Nov 03 22:21:27.192142 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.192095 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dw62d" event={"ID":"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1","Type":"ContainerStarted","Data":"a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a"} Nov 03 22:21:27.193481 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.193455 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4pjvv" event={"ID":"65ac9b7f-736c-405d-babf-089ef215fb8a","Type":"ContainerStarted","Data":"c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2"} Nov 03 22:21:27.366971 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.366927 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:26 +0000 UTC" deadline="2028-05-30 03:23:29.147692334 +0000 UTC" Nov 03 22:21:27.366971 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.366969 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13757h2m1.780727963s" Nov 03 22:21:27.936429 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.936378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:27.936619 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:27.936467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:27.936679 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.936617 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:27.936736 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.936683 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:29.936663374 +0000 UTC m=+6.211343706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:27.936809 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.936752 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:27.936809 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:27.936787 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:29.936774312 +0000 UTC m=+6.211454627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:28.037478 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.037437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:28.037655 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.037636 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:28.037731 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.037661 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:28.037731 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.037676 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:28.037829 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.037756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:30.037735131 +0000 UTC m=+6.312415460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:28.174174 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.174138 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:28.174362 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.174300 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:28.174427 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.174392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:28.174482 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.174465 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:28.174644 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.174622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:28.174769 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:28.174723 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:28.367685 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.367618 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:26 +0000 UTC" deadline="2028-05-26 08:46:24.629108166 +0000 UTC" Nov 03 22:21:28.367685 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:28.367682 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13666h24m56.261430333s" Nov 03 22:21:29.205432 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.203989 2578 generic.go:356] "Generic (PLEG): container finished" podID="b1edcc8793fd6be957384c61df114fef" containerID="ff21a127c9f6c72e3ba05048dbb1a614a5d50123ed131f18469315252c6a5bb0" exitCode=0 Nov 03 22:21:29.205514 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.205363 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerDied","Data":"ff21a127c9f6c72e3ba05048dbb1a614a5d50123ed131f18469315252c6a5bb0"} Nov 03 22:21:29.213711 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.212763 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"6400a77c4e4822387f7b4ea34fe62bb6","Type":"ContainerStarted","Data":"0c925a81512f4ab30658f93038eb69f9e6242bba2b7b33740061d5f4bb1af6cf"} Nov 03 22:21:29.461985 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.461926 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" podStartSLOduration=3.461903892 podStartE2EDuration="3.461903892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:29.460929954 +0000 UTC m=+5.735610293" watchObservedRunningTime="2026-11-03 22:21:29.461903892 +0000 UTC m=+5.736584229" Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.962377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:29.962456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:29.962651 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:29.962717 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:33.962699243 +0000 UTC m=+10.237379582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:29.962792 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:29.962860 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:29.962826 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:33.962814844 +0000 UTC m=+10.237495172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:30.063947 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:30.063695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:30.063947 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.063836 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:30.063947 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.063856 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:30.063947 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.063865 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:30.063947 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.063913 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:34.063898624 +0000 UTC m=+10.338578951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:30.173534 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:30.173183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:30.173534 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.173344 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:30.173534 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:30.173412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:30.173534 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.173483 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:30.174905 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:30.174148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:30.174905 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:30.174268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:30.234793 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:30.234753 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerStarted","Data":"030087c99c1ff5c954c97b99fff6c9234c4adbf9542aedea096b93644b96054b"} Nov 03 22:21:31.245963 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:31.245708 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="60435a2213f641dad2368f0e2e29b429af7b3af34f71219472eaccbcbcf2b518" exitCode=0 Nov 03 22:21:31.247170 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:31.246809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"60435a2213f641dad2368f0e2e29b429af7b3af34f71219472eaccbcbcf2b518"} Nov 03 22:21:31.382937 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:31.381933 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" podStartSLOduration=5.381914758 podStartE2EDuration="5.381914758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:30.363771435 +0000 UTC m=+6.638451775" watchObservedRunningTime="2026-11-03 22:21:31.381914758 +0000 UTC m=+7.656595097" Nov 03 22:21:32.173644 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:32.173608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:32.173850 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:32.173755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:32.174272 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:32.174190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:32.174379 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:32.174310 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:32.174437 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:32.174379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:32.174536 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:32.174512 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:32.253686 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:32.253643 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mlhk9" event={"ID":"661b34e8-d977-46ae-a3c6-6f09a1182f8d","Type":"ContainerStarted","Data":"6e7835c1a7aad168f56bba1f34222cbf02d35a23a51eb4244dff1a0e8332e54f"} Nov 03 22:21:33.103182 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.103098 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:33.103955 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.103745 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:33.230586 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.230518 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mlhk9" podStartSLOduration=3.551385689 podStartE2EDuration="8.230501236s" totalImagesPullingTime="4.679115547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:32.381221419 +0000 UTC m=+8.655901758" watchObservedRunningTime="2026-11-03 22:21:33.230501236 +0000 UTC m=+9.505181574" Nov 03 22:21:33.262175 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.262118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hzqr" event={"ID":"450acb37-2b14-4264-b470-ce0a63835e8b","Type":"ContainerStarted","Data":"44e32531ea2f72bc84c8871200308d18169fe224623354b62d5b0d23b6f43001"} Nov 03 22:21:33.262703 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.262372 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:33.263054 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.263017 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:21:33.519466 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.519294 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6hzqr" podStartSLOduration=2.5461089 podStartE2EDuration="8.519273152s" totalImagesPullingTime="5.973164252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:33.388984583 +0000 UTC m=+9.663664914" watchObservedRunningTime="2026-11-03 22:21:33.519273152 +0000 UTC m=+9.793953491" Nov 03 22:21:33.999778 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.999725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:33.999997 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:33.999820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:33.999997 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:33.999980 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:34.000137 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.000070 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.000031468 +0000 UTC m=+18.274711828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:34.000509 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.000492 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:34.000570 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.000546 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.000530498 +0000 UTC m=+18.275210831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:34.101055 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:34.101001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:34.101268 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.101237 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:34.101268 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.101264 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:34.101375 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.101278 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:34.101375 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.101341 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.101321673 +0000 UTC m=+18.376002011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:34.171718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.171861 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:34.171974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.172085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:34.172145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:34.173177 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:34.172212 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:34.268410 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:34.267834 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" event={"ID":"597ba6ab-ea33-4347-871a-f4edc6188d05","Type":"ContainerStarted","Data":"adb6ed69d9eca7ab7fef8bec3b97c070ee9edc3a66e8bcb1ff442d47cf4b81b6"} Nov 03 22:21:35.779466 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:35.779435 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:21:36.161553 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.161425 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:21:35.779460619Z","UUID":"5c8fbb32-cf34-4e7c-b18b-9dc1a40889aa","Handler":null,"Name":"","Endpoint":""} Nov 03 22:21:36.163588 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.163336 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:21:36.163588 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.163372 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:21:36.173783 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.173753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:36.173931 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:36.173907 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:36.174234 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.174192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:36.174386 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.174365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:36.174508 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:36.174359 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:36.174567 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:36.174513 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:36.275849 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.275766 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" event={"ID":"597ba6ab-ea33-4347-871a-f4edc6188d05","Type":"ContainerStarted","Data":"d93710441f5a32424eda23dc8d006972fb6e29681d891be7fba8be993c1416a7"} Nov 03 22:21:36.278185 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.278130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h85gn" event={"ID":"87a5d53c-512b-48d9-be21-a6f82b3f3761","Type":"ContainerStarted","Data":"ae5aa52208595d1f5bf0eb02ea77c02111c0fdb3c8b99493d6672b1133f2e2b5"} Nov 03 22:21:36.280935 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.280474 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dw62d" event={"ID":"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1","Type":"ContainerStarted","Data":"48d5774b0deb87b9131ec4c415089814ce5545b19df2a5a6864594bf399c0e53"} Nov 03 22:21:36.404479 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.404424 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h85gn" podStartSLOduration=2.392050938 podStartE2EDuration="11.404409889s" totalImagesPullingTime="9.012358951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:36.404188696 +0000 UTC m=+12.678869031" watchObservedRunningTime="2026-11-03 22:21:36.404409889 +0000 UTC m=+12.679090222" Nov 03 22:21:36.520194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:36.520133 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dw62d" podStartSLOduration=2.637106968 podStartE2EDuration="11.520116447s" totalImagesPullingTime="8.883009479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:36.520082847 +0000 UTC m=+12.794763183" watchObservedRunningTime="2026-11-03 22:21:36.520116447 +0000 UTC m=+12.794796778" Nov 03 22:21:37.285598 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:37.285372 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="29f895f7f3795825260f9a794e597a200e4d2ed9845d596c241effb1402c6c33" exitCode=0 Nov 03 22:21:37.286088 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:37.285521 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"29f895f7f3795825260f9a794e597a200e4d2ed9845d596c241effb1402c6c33"} Nov 03 22:21:38.172719 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.172689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:38.172719 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.172710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:38.172719 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.172730 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:38.172967 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:38.172838 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:38.175208 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:38.175175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:38.175341 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:38.175309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:38.289201 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.289167 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4pjvv" event={"ID":"65ac9b7f-736c-405d-babf-089ef215fb8a","Type":"ContainerStarted","Data":"2b16988e4bcfa087eb3ce6bb31594a6e0ae811c1c86392c6178ad9106553be47"} Nov 03 22:21:38.291349 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.291307 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" event={"ID":"597ba6ab-ea33-4347-871a-f4edc6188d05","Type":"ContainerStarted","Data":"f49adc3207e81058641aa0d3a2040c5352a595f1fd7de1b265d41e508d5d5657"} Nov 03 22:21:38.404822 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.404755 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4pjvv" podStartSLOduration=4.23936786 podStartE2EDuration="13.404741584s" totalImagesPullingTime="9.165373724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:38.404678078 +0000 UTC m=+14.679358414" watchObservedRunningTime="2026-11-03 22:21:38.404741584 +0000 UTC m=+14.679421919" Nov 03 22:21:38.525150 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:38.525106 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" podStartSLOduration=2.9021897599999997 podStartE2EDuration="13.525091227s" totalImagesPullingTime="10.622901467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:38.524892426 +0000 UTC m=+14.799572788" watchObservedRunningTime="2026-11-03 22:21:38.525091227 +0000 UTC m=+14.799771563" Nov 03 22:21:39.295889 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:39.295856 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="9a24ecd0534cb1bf73a80fc76ec0f40ba302d5ad0af2dda623afbe322898170d" exitCode=0 Nov 03 22:21:39.296733 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:39.295943 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"9a24ecd0534cb1bf73a80fc76ec0f40ba302d5ad0af2dda623afbe322898170d"} Nov 03 22:21:39.300312 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:39.299869 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" event={"ID":"d6a01977-b5b7-4177-bfd4-e5d2265b442e","Type":"ContainerStarted","Data":"925f99dbd6ec23d1cd0f56f6dc7d7221ceec4595e77f5a9988373f9a54179efa"} Nov 03 22:21:39.594282 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:39.594184 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" podStartSLOduration=2.7874771259999997 podStartE2EDuration="14.594169368s" totalImagesPullingTime="11.806692242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:39.593967963 +0000 UTC m=+15.868648294" watchObservedRunningTime="2026-11-03 22:21:39.594169368 +0000 UTC m=+15.868849704" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:40.172070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:40.172102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:40.172182 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:40.172221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:40.172305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:40.172492 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:40.172365 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:41.307445 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:41.307211 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="bb2b2b5d601a459e4ebc689d344761b2e1e701234f2d892229fbee4aabe71aae" exitCode=0 Nov 03 22:21:41.308036 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:41.307300 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"bb2b2b5d601a459e4ebc689d344761b2e1e701234f2d892229fbee4aabe71aae"} Nov 03 22:21:42.069404 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.069370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:42.069404 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.069414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:42.069673 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.069515 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:42.069673 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.069517 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:42.069673 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.069568 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.069555731 +0000 UTC m=+34.344236047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:42.069673 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.069582 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret podName:71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.069574287 +0000 UTC m=+34.344254599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret") pod "global-pull-secret-syncer-dchqn" (UID: "71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:42.170042 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.170005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:42.170280 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.170175 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:42.170280 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.170196 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:42.170280 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.170205 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cnlnw for pod openshift-network-diagnostics/network-check-target-wmt2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:42.170280 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.170258 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw podName:cda96db7-532a-4209-8761-1ef2ba00260e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.170246128 +0000 UTC m=+34.444926440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cnlnw" (UniqueName: "kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw") pod "network-check-target-wmt2m" (UID: "cda96db7-532a-4209-8761-1ef2ba00260e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:42.172129 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.172107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:42.172228 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.172113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:42.172228 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.172205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:42.172299 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.172258 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:42.172329 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:42.172299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:42.172373 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:42.172358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:43.316072 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.315805 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerStarted","Data":"52764549c4d0cda16ff985d19fd42852c7a0c6140cc6e1896fbd8cb2db4de0af"} Nov 03 22:21:43.319112 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319087 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"545212c87fa6ab28d137d7d091dc5c288787ce0acc8c77692eefe9517ae961b7"} Nov 03 22:21:43.319226 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"ad00fe57bf41f593c68e95d256a013e7442edf6bf0862b6947227c337d295a08"} Nov 03 22:21:43.319226 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319183 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"f95d029b78908fac5627deb3d2210ef03cbab5f1b3e4fdcf06c704c689f0a6d0"} Nov 03 22:21:43.319226 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319210 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"b9e65c0a35029a260405a19ac64154a4645cc8ebee92f90ab7c30ffe99ef4b9f"} Nov 03 22:21:43.319321 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319236 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"f898f8dbe8eb2a90dfdbe54c8b0e54287f25775bf2356919cd9fcf676540b879"} Nov 03 22:21:43.319321 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:43.319262 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"c709a1623d6ea236c93dbeabefc52844cda1e2c1ca7f8c7ab47bca4915e44339"} Nov 03 22:21:44.173023 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:44.172977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:44.173198 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:44.173054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:44.173198 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:44.173069 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:44.173198 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:44.173141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:44.173198 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:44.173175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:44.173325 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:44.173219 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:44.324029 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:44.323970 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="52764549c4d0cda16ff985d19fd42852c7a0c6140cc6e1896fbd8cb2db4de0af" exitCode=0 Nov 03 22:21:44.324449 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:44.324016 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"52764549c4d0cda16ff985d19fd42852c7a0c6140cc6e1896fbd8cb2db4de0af"} Nov 03 22:21:45.328794 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:45.328763 2578 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="0f4fa8058118a5732c91f0cf3b5d107f63f8b5465d31c97b2bda8cf8532be0ab" exitCode=0 Nov 03 22:21:45.329349 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:45.328840 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerDied","Data":"0f4fa8058118a5732c91f0cf3b5d107f63f8b5465d31c97b2bda8cf8532be0ab"} Nov 03 22:21:45.332291 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:45.332272 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"db0687a05b99a027ea6defe7ead2f78cc54edced122dea2e39a6ccdc67fff1ff"} Nov 03 22:21:46.172739 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:46.172695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:46.172739 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:46.172718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:46.172961 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:46.172836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:46.172961 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:46.172862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:46.172961 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:46.172897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:46.173164 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:46.172970 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:46.338148 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:46.338111 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" event={"ID":"66166d99-28a5-4a07-af27-2f605fb566df","Type":"ContainerStarted","Data":"2d78335d31794b926fecb8e4211399c4d4a6cfac49a112cce6ae2c32c8a66a3f"} Nov 03 22:21:46.466284 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:46.466188 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tzg8f" podStartSLOduration=8.643093336 podStartE2EDuration="21.466173526s" totalImagesPullingTime="12.82308019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:46.4657834 +0000 UTC m=+22.740463734" watchObservedRunningTime="2026-11-03 22:21:46.466173526 +0000 UTC m=+22.740853860" Nov 03 22:21:47.343762 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:47.343582 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" event={"ID":"8a8192fe-9bfe-4b5e-ab36-977ddf31533e","Type":"ContainerStarted","Data":"f046a0b99b1578790190acda075019a8a6c6225ee0249a7962e934901e4417bf"} Nov 03 22:21:47.627660 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:47.627606 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" podStartSLOduration=6.98992929 podStartE2EDuration="22.627592949s" totalImagesPullingTime="15.637663659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:47.576353996 +0000 UTC m=+23.851034330" watchObservedRunningTime="2026-11-03 22:21:47.627592949 +0000 UTC m=+23.902273283" Nov 03 22:21:48.172299 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.172269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:48.172452 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.172322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:48.172452 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.172329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:48.172452 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:48.172395 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:48.172569 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:48.172474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:48.172569 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:48.172547 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:48.346205 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.346172 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:48.346646 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.346240 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:48.346646 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.346283 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:48.363068 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.363039 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:48.366634 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:48.366608 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:21:49.454275 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.454244 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dchqn"] Nov 03 22:21:49.454625 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.454377 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:49.454625 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:49.454456 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:49.457102 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.457080 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmt2m"] Nov 03 22:21:49.457196 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.457163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:49.457240 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:49.457226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:49.460143 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.460124 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8g4v"] Nov 03 22:21:49.460223 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:49.460211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:49.460306 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:49.460292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:51.172462 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:51.172425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:51.172462 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:51.172452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:51.172845 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:51.172455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:51.172845 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:51.172533 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:51.172845 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:51.172621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:51.172845 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:51.172716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:53.172439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:53.172401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:53.172439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:53.172440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:53.172972 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:53.172534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8g4v" podUID="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" Nov 03 22:21:53.172972 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:53.172591 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dchqn" podUID="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" Nov 03 22:21:53.172972 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:53.172611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:53.172972 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:53.172686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmt2m" podUID="cda96db7-532a-4209-8761-1ef2ba00260e" Nov 03 22:21:54.526925 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.526896 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeReady" Nov 03 22:21:54.527336 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.527029 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:21:54.643802 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.643764 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z4nhs"] Nov 03 22:21:54.648540 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.648519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:21:54.683214 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.683178 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx"] Nov 03 22:21:54.685492 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.685475 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9j49"] Nov 03 22:21:54.685626 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.685609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.687592 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.687574 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9dn28"] Nov 03 22:21:54.687714 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.687697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:21:54.689836 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.689818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:21:54.694624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.694605 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j"] Nov 03 22:21:54.697009 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.696973 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt"] Nov 03 22:21:54.697097 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.697079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.699906 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.699883 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plm6l"] Nov 03 22:21:54.700057 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.699978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.702513 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.702496 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:21:54.702616 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.702597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.705448 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.705431 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-t8ztc"] Nov 03 22:21:54.705565 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.705548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.711456 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.711429 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh"] Nov 03 22:21:54.711573 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.711553 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.715029 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.715008 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79487cbf4c-klb6d"] Nov 03 22:21:54.715104 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.715061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:21:54.718494 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.717798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.734621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.734585 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv"] Nov 03 22:21:54.737830 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.737488 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-x6mwc"] Nov 03 22:21:54.737830 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.737575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.740276 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.740260 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xgvtf"] Nov 03 22:21:54.740401 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.740382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.743508 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.743490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.745457 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.745435 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:21:54.746057 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.746037 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.746147 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.746119 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.746368 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.746354 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:21:54.749416 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.749400 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h"] Nov 03 22:21:54.749571 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.749548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.753762 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.753741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:21:54.754135 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.754116 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-slgc6"] Nov 03 22:21:54.757887 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757870 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z4nhs"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757897 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757911 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9dn28"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757925 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9j49"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757937 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757950 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:21:54.758010 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.757983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.765052 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.765032 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh"] Nov 03 22:21:54.768331 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.768310 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-t8ztc"] Nov 03 22:21:54.769446 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.769418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4v6\" (UniqueName: \"kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6\") pod \"kas-connection-checker-685895c7c9-z4nhs\" (UID: \"4657e9d9-df67-410c-8154-655ecd80a494\") " pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:21:54.771447 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.771428 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79487cbf4c-klb6d"] Nov 03 22:21:54.804740 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.804714 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv"] Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805207 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805277 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805286 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805295 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805323 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805321 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.805370 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805371 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805380 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805395 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805450 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805455 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805465 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805514 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.805761 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805760 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805791 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805826 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805850 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805877 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805914 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805937 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.806136 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.805971 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.808442 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.808419 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:21:54.813225 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.809484 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.815245 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.814685 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.857714 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.857560 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:21:54.869818 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.869781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.869965 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.869833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.869965 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.869860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmmv\" (UniqueName: \"kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.870109 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.870109 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.870209 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.870209 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.870209 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.870209 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzts\" (UniqueName: \"kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.870409 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.870409 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.870409 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krz9j\" (UniqueName: \"kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.870409 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.870409 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rs7z\" (UniqueName: \"kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z\") pod \"kas-connection-checker-685895c7c9-9dn28\" (UID: \"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985\") " pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4p7\" (UniqueName: \"kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2x8\" (UniqueName: \"kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8\") pod \"network-check-source-6f78d59f99-vcq5h\" (UID: \"d0472439-1511-4476-8d44-ffcc69b07db8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.870653 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxk2\" (UniqueName: \"kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.870920 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.870979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mzh\" (UniqueName: \"kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlz7\" (UniqueName: \"kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.871194 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782wd\" (UniqueName: \"kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd\") pod \"volume-data-source-validator-58c9f5f46d-rsrsh\" (UID: \"274f3045-f615-43d6-8d73-ebf9199ea724\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5jl\" (UniqueName: \"kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdcp9\" (UniqueName: \"kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9\") pod \"kas-connection-checker-685895c7c9-l9j49\" (UID: \"59f23597-a6bb-4e60-a54f-b7411e073675\") " pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:21:54.871406 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.871621 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4v6\" (UniqueName: \"kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6\") pod \"kas-connection-checker-685895c7c9-z4nhs\" (UID: \"4657e9d9-df67-410c-8154-655ecd80a494\") " pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:21:54.871780 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.871780 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.871701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhk27\" (UniqueName: \"kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.873080 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.873056 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:21:54.873255 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.873104 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:21:54.873662 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.873642 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:21:54.874221 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.874202 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slgc6"] Nov 03 22:21:54.907274 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907180 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plm6l"] Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907637 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907673 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907673 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907695 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907695 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907716 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907756 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907748 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907769 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:21:54.907824 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.907781 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bbwr5\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908018 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908043 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908049 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908067 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908114 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rrf2w\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908165 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vc5c5\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908170 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908204 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908230 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908242 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908315 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-697vs\"" Nov 03 22:21:54.908439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.908336 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:21:54.909052 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.909037 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt"] Nov 03 22:21:54.909655 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.909639 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgvtf"] Nov 03 22:21:54.910241 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.910227 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-x6mwc"] Nov 03 22:21:54.910809 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.910792 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h"] Nov 03 22:21:54.935096 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.934955 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4v6\" (UniqueName: \"kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6\") pod \"kas-connection-checker-685895c7c9-z4nhs\" (UID: \"4657e9d9-df67-410c-8154-655ecd80a494\") " pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:21:54.959463 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.959433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:21:54.972624 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4p7\" (UniqueName: \"kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.972712 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2x8\" (UniqueName: \"kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8\") pod \"network-check-source-6f78d59f99-vcq5h\" (UID: \"d0472439-1511-4476-8d44-ffcc69b07db8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:21:54.972712 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.972712 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.972857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.972857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxk2\" (UniqueName: \"kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.972857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.972857 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.973069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.973069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.972932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.973069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.973069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.973307 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.973307 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.973307 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.973307 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mzh\" (UniqueName: \"kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlz7\" (UniqueName: \"kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.973589 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.973777 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.973777 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.973836 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.973777 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.473759144 +0000 UTC m=+31.748439475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:21:54.973836 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.973822 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:54.973836 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.973835 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:21:54.973921 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.973921 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.973891 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.473875707 +0000 UTC m=+31.748556021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:21:54.973983 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.973915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.974039 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.974254 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782wd\" (UniqueName: \"kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd\") pod \"volume-data-source-validator-58c9f5f46d-rsrsh\" (UID: \"274f3045-f615-43d6-8d73-ebf9199ea724\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:21:54.974320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.974320 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5jl\" (UniqueName: \"kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdcp9\" (UniqueName: \"kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9\") pod \"kas-connection-checker-685895c7c9-l9j49\" (UID: \"59f23597-a6bb-4e60-a54f-b7411e073675\") " pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.974581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhk27\" (UniqueName: \"kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmmv\" (UniqueName: \"kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.974760 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.974775 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.974819 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.47480234 +0000 UTC m=+31.749482652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzts\" (UniqueName: \"kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.975083 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.974959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krz9j\" (UniqueName: \"kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rs7z\" (UniqueName: \"kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z\") pod \"kas-connection-checker-685895c7c9-9dn28\" (UID: \"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985\") " pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.975826 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.975665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.976293 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976037 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:21:54.976293 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976099 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.476068706 +0000 UTC m=+31.750749035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:21:54.976293 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976183 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:21:54.976293 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976219 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.476206456 +0000 UTC m=+31.750886771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:21:54.976293 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.976238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:54.976522 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.976347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.976522 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976465 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:54.976522 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976522 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.476507922 +0000 UTC m=+31.751188248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:54.976813 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.976792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.976931 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976870 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:21:54.976931 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.976934 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.476919927 +0000 UTC m=+31.751600242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:21:54.976931 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.976929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.977341 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.977313 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.977457 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.977389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:54.977457 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.977435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:54.977619 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.977598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.977835 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.977737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:54.978076 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.978050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.978170 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.978140 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:21:54.978233 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:54.978194 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:55.478178371 +0000 UTC m=+31.752858705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:21:54.978498 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.978475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.978626 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.978604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.978841 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.978818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:54.978906 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.978853 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:54.979536 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.979502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:54.980411 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.980096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:54.980411 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.980335 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:54.980609 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.980588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:54.980942 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.980921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.981172 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.981149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:54.981317 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.981302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.981375 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.981338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:54.982277 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:54.982255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:55.035266 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.035231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4p7\" (UniqueName: \"kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:55.036058 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.036030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxk2\" (UniqueName: \"kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:55.036736 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.036713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:55.037351 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.037312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:55.037732 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.037708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mzh\" (UniqueName: \"kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:55.038161 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.038134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2x8\" (UniqueName: \"kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8\") pod \"network-check-source-6f78d59f99-vcq5h\" (UID: \"d0472439-1511-4476-8d44-ffcc69b07db8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:21:55.038462 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.038422 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782wd\" (UniqueName: \"kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd\") pod \"volume-data-source-validator-58c9f5f46d-rsrsh\" (UID: \"274f3045-f615-43d6-8d73-ebf9199ea724\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:21:55.041331 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.040660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlz7\" (UniqueName: \"kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:55.043310 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.043280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:55.043397 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.043359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krz9j\" (UniqueName: \"kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:55.044095 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.044070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdcp9\" (UniqueName: \"kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9\") pod \"kas-connection-checker-685895c7c9-l9j49\" (UID: \"59f23597-a6bb-4e60-a54f-b7411e073675\") " pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:21:55.044915 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.044891 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rs7z\" (UniqueName: \"kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z\") pod \"kas-connection-checker-685895c7c9-9dn28\" (UID: \"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985\") " pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:21:55.045323 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.045303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhk27\" (UniqueName: \"kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:55.045625 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.045609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:55.046100 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.046070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzts\" (UniqueName: \"kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:55.046284 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.046259 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmmv\" (UniqueName: \"kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:55.047519 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.047502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5jl\" (UniqueName: \"kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:55.059467 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.059440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:21:55.066889 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.066871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:21:55.105065 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.105027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:21:55.116708 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.116203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:21:55.172202 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.172177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:55.172654 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.172409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:55.172654 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.172421 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:55.228869 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.228834 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:21:55.229022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.228963 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnsfs\"" Nov 03 22:21:55.229022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.228968 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ft6gs\"" Nov 03 22:21:55.229022 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.229012 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:21:55.229157 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.229032 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25tw\"" Nov 03 22:21:55.298840 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.298809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:21:55.305814 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.305796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:21:55.316159 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.316137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:21:55.322678 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.322660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:21:55.339162 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.339139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:21:55.347770 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.347501 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z4nhs"] Nov 03 22:21:55.434462 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.433860 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-t8ztc"] Nov 03 22:21:55.436211 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.435383 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh"] Nov 03 22:21:55.484561 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:55.484676 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:55.484676 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:55.484676 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:55.484676 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:55.484793 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:55.484793 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:55.484793 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.484758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:55.484871 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.484848 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:55.484871 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.484857 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:21:55.484924 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.484897 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.48488515 +0000 UTC m=+32.759565466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:21:55.484960 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.484945 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:21:55.485012 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.484965 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.48495924 +0000 UTC m=+32.759639551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:21:55.485047 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485026 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:21:55.485047 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485046 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.485040049 +0000 UTC m=+32.759720361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:21:55.485113 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485090 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.485083894 +0000 UTC m=+32.759764205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:21:55.485147 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485128 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:55.485147 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485135 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:21:55.485197 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485152 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.48514679 +0000 UTC m=+32.759827102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:21:55.485197 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485184 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:21:55.485258 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485231 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.485196514 +0000 UTC m=+32.759876826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:21:55.485289 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485269 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:21:55.485316 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485304 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.485298006 +0000 UTC m=+32.759978318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:21:55.485346 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485336 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:55.485373 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:55.485355 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:56.485349139 +0000 UTC m=+32.760029452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:55.495115 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.495090 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h"] Nov 03 22:21:55.498018 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.497971 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0472439_1511_4476_8d44_ffcc69b07db8.slice/crio-854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8 WatchSource:0}: Error finding container 854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8: Status 404 returned error can't find the container with id 854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8 Nov 03 22:21:55.521013 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.520960 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slgc6"] Nov 03 22:21:55.523191 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.523168 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d9997c_a4f7_4f1a_8595_5a4e35c4ea65.slice/crio-986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03 WatchSource:0}: Error finding container 986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03: Status 404 returned error can't find the container with id 986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03 Nov 03 22:21:55.676287 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.676147 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx"] Nov 03 22:21:55.678567 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.678541 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d6573e_1446_4ea2_b6f1_9cf9e19a161d.slice/crio-c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf WatchSource:0}: Error finding container c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf: Status 404 returned error can't find the container with id c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf Nov 03 22:21:55.685199 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.685125 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9j49"] Nov 03 22:21:55.688028 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.687984 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f23597_a6bb_4e60_a54f_b7411e073675.slice/crio-6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd WatchSource:0}: Error finding container 6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd: Status 404 returned error can't find the container with id 6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd Nov 03 22:21:55.720688 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.720662 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9dn28"] Nov 03 22:21:55.724689 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.724662 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j"] Nov 03 22:21:55.726890 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.726854 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dae9dcf_d659_41b6_8b8c_8ce5d2f5a985.slice/crio-1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8 WatchSource:0}: Error finding container 1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8: Status 404 returned error can't find the container with id 1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8 Nov 03 22:21:55.730375 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.730332 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f77da67_81d4_4a4e_a92b_c69542057b14.slice/crio-a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0 WatchSource:0}: Error finding container a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0: Status 404 returned error can't find the container with id a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0 Nov 03 22:21:55.759436 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:55.759412 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plm6l"] Nov 03 22:21:55.762567 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:55.762539 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode906de24_2e38_4a47_81cd_581b7c617454.slice/crio-c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235 WatchSource:0}: Error finding container c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235: Status 404 returned error can't find the container with id c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235 Nov 03 22:21:56.367554 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.367503 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" event={"ID":"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d","Type":"ContainerStarted","Data":"c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf"} Nov 03 22:21:56.369503 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.369473 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" event={"ID":"d0472439-1511-4476-8d44-ffcc69b07db8","Type":"ContainerStarted","Data":"854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8"} Nov 03 22:21:56.371484 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.371453 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" event={"ID":"4657e9d9-df67-410c-8154-655ecd80a494","Type":"ContainerStarted","Data":"dad959dae1c67beb5b8ecd34f1c68ebf04e9ecd6824ad54b11e75ad5f302be26"} Nov 03 22:21:56.371591 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.371497 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" event={"ID":"4657e9d9-df67-410c-8154-655ecd80a494","Type":"ContainerStarted","Data":"216f1639a8b643850f6470f0e6150e0080a5f0c15857f2ef5ed7ed6fcfe7265a"} Nov 03 22:21:56.373300 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.373277 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plm6l" event={"ID":"e906de24-2e38-4a47-81cd-581b7c617454","Type":"ContainerStarted","Data":"c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235"} Nov 03 22:21:56.375040 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.375016 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" event={"ID":"6f77da67-81d4-4a4e-a92b-c69542057b14","Type":"ContainerStarted","Data":"a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0"} Nov 03 22:21:56.376472 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.376453 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" event={"ID":"274f3045-f615-43d6-8d73-ebf9199ea724","Type":"ContainerStarted","Data":"796fb93d548cbf4d5a6c964ef420d5b064a0a4109175a06751fe17a3ddca5308"} Nov 03 22:21:56.378437 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.378416 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9dn28" event={"ID":"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985","Type":"ContainerStarted","Data":"81fcf9f6deb210844aff98b81ca99fde4c41cd123b407b20ee1530f55171692f"} Nov 03 22:21:56.378529 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.378459 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9dn28" event={"ID":"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985","Type":"ContainerStarted","Data":"1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8"} Nov 03 22:21:56.380297 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.380275 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l9j49" event={"ID":"59f23597-a6bb-4e60-a54f-b7411e073675","Type":"ContainerStarted","Data":"713d9c58d33286133d24d0d0da73a79827fc34c49cc031099218162e33d68ac2"} Nov 03 22:21:56.380391 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.380316 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l9j49" event={"ID":"59f23597-a6bb-4e60-a54f-b7411e073675","Type":"ContainerStarted","Data":"6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd"} Nov 03 22:21:56.382158 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.382136 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slgc6" event={"ID":"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65","Type":"ContainerStarted","Data":"986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03"} Nov 03 22:21:56.383473 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.383455 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerStarted","Data":"b8fb93ab2863fb1433f87038aac7b1d087ab87a0f2f9ab2b41cdfcd67112aa96"} Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.501752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.501810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.501841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.501919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.501922 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.501956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.501982 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.501965341 +0000 UTC m=+34.776645670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502044 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502055 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502089 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502077747 +0000 UTC m=+34.776758078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.502051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502105 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502115 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.502123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502145 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502135889 +0000 UTC m=+34.776816201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502181 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:56.502408 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.502188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502211 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502201732 +0000 UTC m=+34.776882058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502243 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502271 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.50226253 +0000 UTC m=+34.776942856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502294 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502325 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502316095 +0000 UTC m=+34.776996425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502339 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502357627 +0000 UTC m=+34.777037942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:21:56.503335 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:56.502386 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:58.502377783 +0000 UTC m=+34.777058110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:21:56.927579 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:56.926078 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9dn28" podStartSLOduration=413.926059569 podStartE2EDuration="6m53.926059569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:56.925466927 +0000 UTC m=+33.200147260" watchObservedRunningTime="2026-11-03 22:21:56.926059569 +0000 UTC m=+33.200739906" Nov 03 22:21:57.162263 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:57.162200 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" podStartSLOduration=414.162179153 podStartE2EDuration="6m54.162179153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:57.044343819 +0000 UTC m=+33.319024154" watchObservedRunningTime="2026-11-03 22:21:57.162179153 +0000 UTC m=+33.436859489" Nov 03 22:21:58.125126 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.123796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:58.125126 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.123963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:21:58.125126 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.124110 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:21:58.125126 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.124185 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:30.124166698 +0000 UTC m=+66.398847035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : secret "metrics-daemon-secret" not found Nov 03 22:21:58.133248 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.133218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:58.193435 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.193393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:21:58.225477 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.225441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:58.228701 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.228669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:58.395204 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.394448 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" event={"ID":"274f3045-f615-43d6-8d73-ebf9199ea724","Type":"ContainerStarted","Data":"ae70ba6ab255be912a16ae9328507693156a077db255b2684cafbdf6483228a1"} Nov 03 22:21:58.500817 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.500522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:21:58.513192 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.513137 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l9j49" podStartSLOduration=415.513114986 podStartE2EDuration="6m55.513114986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:57.163006388 +0000 UTC m=+33.437686721" watchObservedRunningTime="2026-11-03 22:21:58.513114986 +0000 UTC m=+34.787795321" Nov 03 22:21:58.513956 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.513913 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" podStartSLOduration=405.904990988 podStartE2EDuration="6m48.513898336s" totalImagesPullingTime="2.608907348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:58.512194081 +0000 UTC m=+34.786874416" watchObservedRunningTime="2026-11-03 22:21:58.513898336 +0000 UTC m=+34.788578671" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.527969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.528298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.528549 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.528617 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.528596523 +0000 UTC m=+38.803276860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529129 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529112554 +0000 UTC m=+38.803792866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529209 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529242 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529296 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529283568 +0000 UTC m=+38.803963882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529354 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:21:58.529758 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529363 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529392 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529382645 +0000 UTC m=+38.804062958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529452 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529477122 +0000 UTC m=+38.804157440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529535 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529562 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529553337 +0000 UTC m=+38.804233649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529612 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529641 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529632484 +0000 UTC m=+38.804312802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529689 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:21:58.530622 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:21:58.529719 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.529708986 +0000 UTC m=+38.804389304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:21:58.626199 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.625709 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dchqn"] Nov 03 22:21:58.628936 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:58.628555 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c0cee1_1b74_43c7_8bd2_e9ec84ddb0bb.slice/crio-78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e WatchSource:0}: Error finding container 78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e: Status 404 returned error can't find the container with id 78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e Nov 03 22:21:58.903485 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:58.903459 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmt2m"] Nov 03 22:21:58.908441 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:21:58.908234 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda96db7_532a_4209_8761_1ef2ba00260e.slice/crio-adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3 WatchSource:0}: Error finding container adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3: Status 404 returned error can't find the container with id adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3 Nov 03 22:21:59.402176 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:59.402093 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmt2m" event={"ID":"cda96db7-532a-4209-8761-1ef2ba00260e","Type":"ContainerStarted","Data":"adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3"} Nov 03 22:21:59.405200 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:21:59.405138 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dchqn" event={"ID":"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb","Type":"ContainerStarted","Data":"78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e"} Nov 03 22:22:00.412231 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:00.411195 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/0.log" Nov 03 22:22:00.412231 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:00.411252 2578 generic.go:356] "Generic (PLEG): container finished" podID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerID="168209216d040d3e1dbaa4f099b5c5fef6eb081c0b8a8fcd3f56ac19ef02cb77" exitCode=255 Nov 03 22:22:00.412231 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:00.411287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerDied","Data":"168209216d040d3e1dbaa4f099b5c5fef6eb081c0b8a8fcd3f56ac19ef02cb77"} Nov 03 22:22:00.412231 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:00.411628 2578 scope.go:122] "RemoveContainer" containerID="168209216d040d3e1dbaa4f099b5c5fef6eb081c0b8a8fcd3f56ac19ef02cb77" Nov 03 22:22:01.419965 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.419214 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" event={"ID":"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d","Type":"ContainerStarted","Data":"7d9cf633c8f8b07bbcdd84b62cf5c948f0881cf2b9a00aba610361ba0b9da0bc"} Nov 03 22:22:01.424483 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.423969 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plm6l" event={"ID":"e906de24-2e38-4a47-81cd-581b7c617454","Type":"ContainerStarted","Data":"1afa1b962c64694572f82b3f3d72cc113e056dbc8da0af3978b19cf7b5d1f499"} Nov 03 22:22:01.429146 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.428560 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slgc6" event={"ID":"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65","Type":"ContainerStarted","Data":"584014efad96cd55538e958e015a194ce26725a2323cc5301f98775a35ce2ae5"} Nov 03 22:22:01.433015 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.432921 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:01.434537 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.434159 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/0.log" Nov 03 22:22:01.434537 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.434198 2578 generic.go:356] "Generic (PLEG): container finished" podID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" exitCode=255 Nov 03 22:22:01.434537 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.434241 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerDied","Data":"11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a"} Nov 03 22:22:01.434537 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.434290 2578 scope.go:122] "RemoveContainer" containerID="168209216d040d3e1dbaa4f099b5c5fef6eb081c0b8a8fcd3f56ac19ef02cb77" Nov 03 22:22:01.435549 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.435528 2578 scope.go:122] "RemoveContainer" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" Nov 03 22:22:01.435780 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:01.435755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:01.540698 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.540634 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" podStartSLOduration=409.605640027 podStartE2EDuration="6m54.540613997s" totalImagesPullingTime="4.93497397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:01.539032254 +0000 UTC m=+37.813712589" watchObservedRunningTime="2026-11-03 22:22:01.540613997 +0000 UTC m=+37.815294334" Nov 03 22:22:01.658157 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.658100 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-plm6l" podStartSLOduration=386.107417334 podStartE2EDuration="6m30.658084998s" totalImagesPullingTime="4.550667664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:01.65804363 +0000 UTC m=+37.932723960" watchObservedRunningTime="2026-11-03 22:22:01.658084998 +0000 UTC m=+37.932765331" Nov 03 22:22:01.890950 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.890885 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-slgc6" podStartSLOduration=2.262242532 podStartE2EDuration="7.890865517s" totalImagesPullingTime="5.628622985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:01.777643733 +0000 UTC m=+38.052324066" watchObservedRunningTime="2026-11-03 22:22:01.890865517 +0000 UTC m=+38.165545853" Nov 03 22:22:01.891172 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.891143 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88khg"] Nov 03 22:22:01.913234 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.912831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.952038 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.951977 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88khg"] Nov 03 22:22:01.963373 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.963337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.963527 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.963390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.963587 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.963546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.963654 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.963593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5jl\" (UniqueName: \"kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.963785 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.963764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:01.966199 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.966173 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:01.966376 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.966356 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:01.966376 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:01.966368 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-94lr2\"" Nov 03 22:22:02.065357 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.065313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.065550 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.065437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.065550 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.065469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.065652 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.065549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.065652 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.065575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5jl\" (UniqueName: \"kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.066122 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.066072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.066249 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.066152 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:02.066249 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.066212 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.566192129 +0000 UTC m=+38.840872454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:02.066371 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.066296 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:02.066371 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.066347 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:02.566322277 +0000 UTC m=+38.841002604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:02.066702 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.066675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.130450 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.130416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5jl\" (UniqueName: \"kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.140470 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.140437 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:22:02.166644 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.166576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.201196 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.201165 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:22:02.220832 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.220798 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cbh97\"" Nov 03 22:22:02.269483 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.268407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.269483 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.268458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.269483 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.268498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.269483 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.268548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.369243 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.369209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.369406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.369652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.369689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.369823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.370442 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.371081 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.371031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.434134 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.434066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.440179 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.440154 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:02.440577 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.440561 2578 scope.go:122] "RemoveContainer" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" Nov 03 22:22:02.440830 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.440788 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:02.442443 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.442172 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" event={"ID":"d0472439-1511-4476-8d44-ffcc69b07db8","Type":"ContainerStarted","Data":"803568c28f429c096aa6e1208dbd24dd31c3037ac2c0418e0bbba0ac38989756"} Nov 03 22:22:02.499790 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.499759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:02.572192 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.572372 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:02.572372 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572304 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:02.572372 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572331 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:02.572372 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:02.572372 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:03.572348185 +0000 UTC m=+39.847028497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572394 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:03.572384977 +0000 UTC m=+39.847065297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572424 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572471 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572459416 +0000 UTC m=+46.847139763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572472 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572527 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572551 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572540593 +0000 UTC m=+46.847220909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572558139 +0000 UTC m=+46.847238452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:02.572639 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572729 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572716927 +0000 UTC m=+46.847397254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572768 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572779 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.572792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572807 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572821 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572846 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572835414 +0000 UTC m=+46.847515739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572866 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572852786 +0000 UTC m=+46.847533103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572888 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572907 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572922 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572911942 +0000 UTC m=+46.847592285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:22:02.573238 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:02.572938 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.572928574 +0000 UTC m=+46.847608924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:02.626303 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.625046 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh"] Nov 03 22:22:02.648956 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.647953 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd"] Nov 03 22:22:02.649140 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.649114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:02.663069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.661574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:02.673581 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.673552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlg6\" (UniqueName: \"kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6\") pod \"migrator-66df5d496b-8t2kh\" (UID: \"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:02.682025 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.681935 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" podStartSLOduration=378.272782578 podStartE2EDuration="6m24.681915965s" totalImagesPullingTime="6.409133387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:02.67856609 +0000 UTC m=+38.953246425" watchObservedRunningTime="2026-11-03 22:22:02.681915965 +0000 UTC m=+38.956596302" Nov 03 22:22:02.684613 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.684558 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh"] Nov 03 22:22:02.702850 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.696345 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd"] Nov 03 22:22:02.703197 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.703095 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:22:02.703197 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.703168 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:02.703197 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.703174 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-blfxv\"" Nov 03 22:22:02.775142 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.775103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlg6\" (UniqueName: \"kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6\") pod \"migrator-66df5d496b-8t2kh\" (UID: \"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:02.775298 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.775191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmn4\" (UniqueName: \"kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4\") pod \"migrator-66df5d496b-ggkcd\" (UID: \"362a27d3-e34b-4d55-996c-ad51b6423b50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:02.837507 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.837316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlg6\" (UniqueName: \"kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6\") pod \"migrator-66df5d496b-8t2kh\" (UID: \"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:02.876852 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.876655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmn4\" (UniqueName: \"kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4\") pod \"migrator-66df5d496b-ggkcd\" (UID: \"362a27d3-e34b-4d55-996c-ad51b6423b50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:02.938452 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.938378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmn4\" (UniqueName: \"kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4\") pod \"migrator-66df5d496b-ggkcd\" (UID: \"362a27d3-e34b-4d55-996c-ad51b6423b50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:02.972500 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.972464 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:22:02.982439 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:02.982410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:03.004002 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.003969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:03.385834 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.385750 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh"] Nov 03 22:22:03.389482 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:22:03.389454 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ea7495_41a1_4f01_9b7e_c6ec6cf46097.slice/crio-4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f WatchSource:0}: Error finding container 4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f: Status 404 returned error can't find the container with id 4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f Nov 03 22:22:03.436307 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.436218 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd"] Nov 03 22:22:03.438947 ip-10-0-135-87 kubenswrapper[2578]: W1103 22:22:03.438924 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362a27d3_e34b_4d55_996c_ad51b6423b50.slice/crio-c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184 WatchSource:0}: Error finding container c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184: Status 404 returned error can't find the container with id c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184 Nov 03 22:22:03.446538 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.446496 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" event={"ID":"6f77da67-81d4-4a4e-a92b-c69542057b14","Type":"ContainerStarted","Data":"fce479c91d88ce27080ab8358a108e2ea59c2d46feda21fd8833d405fdb4a100"} Nov 03 22:22:03.448091 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.448062 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmt2m" event={"ID":"cda96db7-532a-4209-8761-1ef2ba00260e","Type":"ContainerStarted","Data":"757fcf1ad5db1a589656976de2ed2290fc586b1806a725e2c8a75471aaa74913"} Nov 03 22:22:03.448196 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.448137 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:22:03.449207 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.449185 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" event={"ID":"362a27d3-e34b-4d55-996c-ad51b6423b50","Type":"ContainerStarted","Data":"c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184"} Nov 03 22:22:03.450215 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.450195 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" event={"ID":"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097","Type":"ContainerStarted","Data":"4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f"} Nov 03 22:22:03.451574 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.451552 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" event={"ID":"e57fcd21-a810-4e2f-880b-d99b80541c6a","Type":"ContainerStarted","Data":"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a"} Nov 03 22:22:03.451656 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.451592 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" event={"ID":"e57fcd21-a810-4e2f-880b-d99b80541c6a","Type":"ContainerStarted","Data":"9b4e6a86c08e1d39eb96169894e5e1a3fd42581a4b6741c8ee47e24ee2d436ce"} Nov 03 22:22:03.564308 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.564242 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" podStartSLOduration=408.512628284 podStartE2EDuration="6m55.564224175s" totalImagesPullingTime="7.051595891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:03.563667768 +0000 UTC m=+39.838348104" watchObservedRunningTime="2026-11-03 22:22:03.564224175 +0000 UTC m=+39.838904512" Nov 03 22:22:03.582570 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.582520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:03.582570 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.582573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:03.582784 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:03.582655 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:03.582784 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:03.582683 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:03.582784 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:03.582727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:05.582706069 +0000 UTC m=+41.857386390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:03.582962 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:03.582863 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:05.582850818 +0000 UTC m=+41.857531130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:03.683605 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:03.683500 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" podStartSLOduration=1.683482683 podStartE2EDuration="1.683482683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:03.6808739 +0000 UTC m=+39.955554236" watchObservedRunningTime="2026-11-03 22:22:03.683482683 +0000 UTC m=+39.958163018" Nov 03 22:22:05.060098 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.060049 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:05.060558 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.060133 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:05.060615 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.060594 2578 scope.go:122] "RemoveContainer" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" Nov 03 22:22:05.060846 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:05.060817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:05.421171 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.421120 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wmt2m" podStartSLOduration=36.736494844 podStartE2EDuration="40.421106234s" totalImagesPullingTime="3.68461139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:03.798832524 +0000 UTC m=+40.073512859" watchObservedRunningTime="2026-11-03 22:22:05.421106234 +0000 UTC m=+41.695786567" Nov 03 22:22:05.422168 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.422151 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8s2f2"] Nov 03 22:22:05.442850 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.442816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.481959 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.481930 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8s2f2"] Nov 03 22:22:05.503546 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.503520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzghf\" (UniqueName: \"kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.503669 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.503639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.503748 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.503735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.531595 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.531569 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:22:05.531595 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.531572 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:22:05.547040 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.546984 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:22:05.547040 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.547018 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:22:05.547200 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.547049 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9kg8s\"" Nov 03 22:22:05.604385 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.604353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.604536 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.604421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzghf\" (UniqueName: \"kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.604626 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.604605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:05.604673 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.604643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.604673 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.604667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:05.604769 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:05.604740 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:05.604819 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:05.604806 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:05.604866 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:05.604834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.604810331 +0000 UTC m=+45.879490645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:05.604866 ip-10-0-135-87 kubenswrapper[2578]: E1103 22:22:05.604853 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.604843008 +0000 UTC m=+45.879523338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:05.605257 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.605234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.606910 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.606892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.666464 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.666435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzghf\" (UniqueName: \"kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:05.767069 ip-10-0-135-87 kubenswrapper[2578]: I1103 22:22:05.766976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:06.183867 ip-10-0-135-87 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:22:06.207403 ip-10-0-135-87 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:22:06.207581 ip-10-0-135-87 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:22:06.207759 ip-10-0-135-87 systemd[1]: kubelet.service: Consumed 3.264s CPU time. Nov 03 22:22:06.216500 ip-10-0-135-87 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:22:06.310838 ip-10-0-135-87 kubenswrapper[5051]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:06.310838 ip-10-0-135-87 kubenswrapper[5051]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:22:06.310838 ip-10-0-135-87 kubenswrapper[5051]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:06.310838 ip-10-0-135-87 kubenswrapper[5051]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:06.313080 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.312966 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.313080 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313079 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.313080 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313082 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.313080 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313086 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313088 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313091 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313094 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313097 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313099 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313102 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313105 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313107 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313110 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313113 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313115 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313118 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313120 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313122 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313125 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313127 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313130 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313132 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.313211 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313135 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313138 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313141 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313143 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313152 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313156 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313158 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313161 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313164 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313167 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313169 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313172 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313174 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313177 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313179 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313182 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313184 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313190 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313194 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313197 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.313672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313201 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313204 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313207 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313210 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313212 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313215 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313217 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313221 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313224 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313227 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313230 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313232 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313235 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313237 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313240 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313242 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313245 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313249 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313251 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.314200 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313254 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313257 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313259 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313262 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313264 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313267 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313270 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313272 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313275 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313277 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313280 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313283 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313286 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313289 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313291 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313294 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313296 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313299 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313307 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313310 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.314816 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313313 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313315 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313318 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313321 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313324 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313326 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313329 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313331 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313334 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313337 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313358 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313363 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313366 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313369 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313372 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313375 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313379 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313383 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313387 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.315331 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313389 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313392 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313395 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313397 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313401 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.313403 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314216 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314224 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314227 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314230 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314233 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314236 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314238 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314241 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314243 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314246 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314249 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314251 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314256 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314260 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.315838 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314263 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314265 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314268 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314271 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314273 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314277 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314279 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314282 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314285 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314288 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314291 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314294 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314296 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314299 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314301 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314304 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314306 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314309 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314312 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314315 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.316357 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314318 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314320 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314323 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314326 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314330 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314334 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314337 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314340 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314343 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314346 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314349 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314351 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314354 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314356 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314359 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314361 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314364 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314366 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314372 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.316837 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314375 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314378 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314381 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314383 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314386 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314388 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314391 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314393 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314396 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314398 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314401 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314407 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314409 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314412 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314414 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314417 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314419 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314422 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314424 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314427 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.317359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314430 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314433 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314435 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314438 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314440 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314442 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314445 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314448 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314450 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314452 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314455 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314457 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314465 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314468 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314471 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314474 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314476 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314479 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314482 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.317882 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314484 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314487 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314489 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314492 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314496 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314499 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314501 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314504 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314507 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314510 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314512 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314515 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314517 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.314520 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315076 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315082 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315085 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315088 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315091 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.318359 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315094 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315096 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315099 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315102 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315104 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315107 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315110 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315113 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315118 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315120 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315123 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315127 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315129 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315132 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315134 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315137 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315139 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315143 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315146 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.318854 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315149 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315151 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315154 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315156 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315159 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315161 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315164 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315168 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315172 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315175 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315178 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315181 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315183 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315185 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315188 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315190 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315193 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315196 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315198 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315200 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.319360 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315206 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315210 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315213 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315216 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315218 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315222 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315226 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315229 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315232 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315235 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315237 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315240 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315243 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315246 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315248 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315251 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315253 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315255 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315258 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.319845 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315261 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315264 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315266 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315269 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315271 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315274 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315276 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315278 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315281 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315283 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315286 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315288 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315290 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315293 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315297 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315300 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315303 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315306 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315309 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.320329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315311 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315314 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315317 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315319 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315321 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315324 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315327 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315329 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315332 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315334 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315337 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315340 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315343 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315345 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315348 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315350 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315353 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315355 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315357 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.320791 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315360 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315363 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315365 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315368 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.315370 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315472 5051 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315484 5051 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315493 5051 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315500 5051 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315506 5051 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315510 5051 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315514 5051 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315518 5051 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315521 5051 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315525 5051 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315528 5051 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315531 5051 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315534 5051 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315537 5051 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315540 5051 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315543 5051 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315546 5051 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315551 5051 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315555 5051 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:22:06.321277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315558 5051 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315561 5051 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315564 5051 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315568 5051 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315571 5051 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315575 5051 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315578 5051 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315581 5051 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315584 5051 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315587 5051 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315590 5051 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315593 5051 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315596 5051 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315600 5051 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315603 5051 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315605 5051 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315610 5051 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315613 5051 flags.go:64] FLAG: --enable-server="true" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315617 5051 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315622 5051 flags.go:64] FLAG: --event-burst="100" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315625 5051 flags.go:64] FLAG: --event-qps="50" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315628 5051 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315631 5051 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315634 5051 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:22:06.321853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315638 5051 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315641 5051 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315644 5051 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315647 5051 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315651 5051 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315653 5051 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315658 5051 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315661 5051 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315664 5051 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315667 5051 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315670 5051 flags.go:64] FLAG: --feature-gates="" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315674 5051 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315677 5051 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315681 5051 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315684 5051 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315687 5051 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315690 5051 flags.go:64] FLAG: --help="false" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315692 5051 flags.go:64] FLAG: --hostname-override="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315696 5051 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315699 5051 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315701 5051 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315705 5051 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315708 5051 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:22:06.322511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315711 5051 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315715 5051 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315718 5051 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315721 5051 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315724 5051 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315727 5051 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315730 5051 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315733 5051 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315735 5051 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315738 5051 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315741 5051 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315744 5051 flags.go:64] FLAG: --lock-file="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315747 5051 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315749 5051 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315753 5051 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315760 5051 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315763 5051 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315765 5051 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315769 5051 flags.go:64] FLAG: --logging-format="text" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315771 5051 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315775 5051 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315778 5051 flags.go:64] FLAG: --manifest-url="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315781 5051 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315785 5051 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315788 5051 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:22:06.323487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315792 5051 flags.go:64] FLAG: --max-pods="110" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315795 5051 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315798 5051 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315800 5051 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315803 5051 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315806 5051 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315809 5051 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315812 5051 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315822 5051 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315825 5051 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315827 5051 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315830 5051 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315834 5051 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315836 5051 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315839 5051 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315842 5051 flags.go:64] FLAG: --port="10250" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315845 5051 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315849 5051 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0aabb26406964e5b8" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315852 5051 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315855 5051 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315858 5051 flags.go:64] FLAG: --register-node="true" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315861 5051 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315866 5051 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315869 5051 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315872 5051 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315875 5051 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:22:06.324219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315879 5051 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315881 5051 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315885 5051 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315888 5051 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315891 5051 flags.go:64] FLAG: --runonce="false" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315893 5051 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315896 5051 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315900 5051 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315902 5051 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315905 5051 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315909 5051 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315912 5051 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315915 5051 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315917 5051 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315922 5051 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315924 5051 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315927 5051 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315930 5051 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315933 5051 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315936 5051 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315941 5051 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315944 5051 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315947 5051 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315951 5051 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315954 5051 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:22:06.324844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315956 5051 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315959 5051 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315962 5051 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315966 5051 flags.go:64] FLAG: --v="2" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315970 5051 flags.go:64] FLAG: --version="false" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315975 5051 flags.go:64] FLAG: --vmodule="" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315979 5051 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.315982 5051 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316133 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316138 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316141 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316143 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316146 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316148 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316151 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316153 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316156 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316158 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316161 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316164 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316166 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.325461 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316170 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316173 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316176 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316178 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316181 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316183 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316186 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316189 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316191 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316194 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316196 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316199 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316201 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316204 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316208 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316211 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316215 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316218 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316222 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.325953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316225 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316229 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316232 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316234 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316237 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316240 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316243 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316245 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316247 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316250 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316252 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316254 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316257 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316260 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316263 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316265 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316268 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316270 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316273 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316276 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.326446 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316280 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316282 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316285 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316287 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316290 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316292 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316294 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316298 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316301 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316304 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316306 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316309 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316311 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316314 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316317 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316319 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316322 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316324 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316327 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316329 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.326925 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316332 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316335 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316338 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316340 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316342 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316346 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316349 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316351 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316354 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316356 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316359 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316362 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316364 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316367 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316370 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316374 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316378 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316381 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316384 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.327509 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316388 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316390 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316393 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316396 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316398 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316401 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316403 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316406 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316408 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316411 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316413 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316416 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316418 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316421 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316423 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.316450 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:06.328058 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316494 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316499 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316503 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316506 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316508 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316511 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316513 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316516 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316519 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316522 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316524 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316527 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316530 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316532 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316535 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316537 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316540 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316543 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316546 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.328469 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316549 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316551 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316553 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316556 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316558 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316561 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316563 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316565 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316568 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316570 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316573 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316575 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316578 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316580 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316583 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316586 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316590 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316593 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316596 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.328982 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316599 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316601 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316604 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316606 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316608 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316611 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316614 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316617 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316619 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316622 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316624 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316628 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316630 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316633 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316635 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316639 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316642 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316645 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316648 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316650 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.329577 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316653 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316655 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316658 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316660 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316663 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316665 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316668 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316671 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316673 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316675 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316678 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316680 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316683 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316685 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316687 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316690 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316692 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316695 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316697 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316700 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.330221 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316703 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316706 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316708 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316712 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316716 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316719 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316722 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316725 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316728 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316731 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316734 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316737 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316739 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316742 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316744 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316747 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316749 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316752 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316754 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.330740 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316757 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316759 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316762 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316764 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316767 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316769 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316772 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316775 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.316777 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.321030 5051 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.321047 5051 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.321066 5051 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.321073 5051 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321149 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321155 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321158 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321161 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321165 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321167 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321170 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.331275 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321173 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321176 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321179 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321182 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321184 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321187 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321190 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321193 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321195 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321198 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321201 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321204 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321206 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321209 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321211 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321214 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321217 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321220 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321223 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.331754 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321225 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321228 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321230 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321233 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321236 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321239 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321255 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321258 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321262 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321267 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321270 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321274 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321277 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321280 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321282 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321285 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321288 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321291 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321294 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.332224 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321297 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321299 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321302 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321304 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321307 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321309 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321312 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321314 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321317 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321319 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321322 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321325 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321327 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321330 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321332 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321334 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321337 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321340 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321342 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321345 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.332684 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321353 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321356 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321358 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321360 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321363 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321365 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321367 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321370 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321373 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321376 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321378 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321381 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321384 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321386 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321389 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321391 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321393 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321396 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321398 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321401 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.333345 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321403 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321406 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321409 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321411 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321413 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321423 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321426 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321429 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321431 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321436 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321439 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321442 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321444 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321453 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321455 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321458 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321461 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321463 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321466 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.333820 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321468 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321470 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.321500 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321692 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321696 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321699 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321702 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321705 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321707 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321710 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321712 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321714 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321717 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321719 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321722 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321725 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.334289 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321728 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321730 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321733 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321736 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321738 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321741 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321743 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321746 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321748 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321751 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321754 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321763 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321766 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321768 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321771 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321773 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321776 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321778 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321781 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321783 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.334672 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321786 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321789 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321792 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321794 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321797 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321799 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321802 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321804 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321806 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321808 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321811 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321814 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321817 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321819 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321821 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321824 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321826 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321828 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321831 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.335171 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321833 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321836 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321838 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321840 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321843 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321851 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321853 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321856 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321859 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321861 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321864 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321866 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321870 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321873 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321877 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321879 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321882 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321885 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321887 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.335686 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321889 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321892 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321894 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321896 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321898 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321901 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321903 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321906 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321908 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321911 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321913 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321916 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321918 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321921 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321923 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321926 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321928 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321931 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321933 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.336225 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321942 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321947 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321950 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321952 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321955 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321957 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321960 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321962 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321965 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321967 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321970 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321972 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321974 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321977 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321979 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.321981 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.336732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.322018 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322064 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322069 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322073 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322077 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322080 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322083 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322086 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322091 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322094 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322097 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322099 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322101 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322104 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322106 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322109 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322112 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322120 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322123 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:06.337198 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322125 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322128 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322131 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322134 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322136 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322139 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322141 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322144 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322146 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322150 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322154 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322157 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322160 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322163 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322166 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322168 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322171 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322173 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322176 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:06.337698 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322178 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322181 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322184 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322187 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322190 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322192 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322200 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322203 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322206 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322208 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322211 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322213 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322274 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322464 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322475 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322485 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322495 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322503 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322515 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322525 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:06.338192 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322533 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322543 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322552 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322560 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322570 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322580 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322595 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322607 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322616 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322625 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322632 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322639 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322646 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322654 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322666 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322674 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322680 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322694 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322701 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322709 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:06.338724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322716 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322724 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322732 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322738 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322745 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322831 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322840 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322847 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322854 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322871 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322878 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322886 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322893 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322900 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322907 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322913 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322921 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322928 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322956 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:06.339329 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322962 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322966 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322970 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322979 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.322984 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.323004 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.323008 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.323015 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.323020 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:06.323024 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.323158 5051 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.324794 5051 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.324865 5051 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.325081 5051 server.go:1014] "Starting client certificate rotation" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.325219 5051 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.325274 5051 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:25 +0000 UTC" deadline="2028-07-19 02:30:37.501853168 +0000 UTC" Nov 03 22:22:06.339781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.325309 5051 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14956h8m31.176546514s" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.325580 5051 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.326787 5051 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.328055 5051 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.329790 5051 log.go:25] "Validated CRI v1 image API" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.330541 5051 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:22:06.340255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.332562 5051 fs.go:135] Filesystem UUIDs: map[48a23b4c-f1cf-4b70-8724-37ccdfa68e72:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d3c1b729-ee41-4203-bbbf-b276f817e818:/dev/nvme0n1p4] Nov 03 22:22:06.340477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.332584 5051 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/216f1639a8b643850f6470f0e6150e0080a5f0c15857f2ef5ed7ed6fcfe7265a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/216f1639a8b643850f6470f0e6150e0080a5f0c15857f2ef5ed7ed6fcfe7265a/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30c152cb9d7ddbd2848b5a4e0266d4e06c45d8503af62670f94a7718a41cefd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30c152cb9d7ddbd2848b5a4e0266d4e06c45d8503af62670f94a7718a41cefd0/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/796fb93d548cbf4d5a6c964ef420d5b064a0a4109175a06751fe17a3ddca5308/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/796fb93d548cbf4d5a6c964ef420d5b064a0a4109175a06751fe17a3ddca5308/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b4e6a86c08e1d39eb96169894e5e1a3fd42581a4b6741c8ee47e24ee2d436ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b4e6a86c08e1d39eb96169894e5e1a3fd42581a4b6741c8ee47e24ee2d436ce/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ebf34f0bf961d7ea1bdb48a89f7b2171ce0dc1b5b9fc65aa046854179798295/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ebf34f0bf961d7ea1bdb48a89f7b2171ce0dc1b5b9fc65aa046854179798295/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8fb93ab2863fb1433f87038aac7b1d087ab87a0f2f9ab2b41cdfcd67112aa96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8fb93ab2863fb1433f87038aac7b1d087ab87a0f2f9ab2b41cdfcd67112aa96/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d945ded0fa94f63c8ee76ce2a3595a8e5d9f121395a413551b786bf9afdd178f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d945ded0fa94f63c8ee76ce2a3595a8e5d9f121395a413551b786bf9afdd178f/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~projected/kube-api-access-krz9j:{mountpoint:/var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~projected/kube-api-access-krz9j major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~projected/kube-api-access-pj5jl:{mountpoint:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~projected/kube-api-access-pj5jl major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274f3045-f615-43d6-8d73-ebf9199ea724/volumes/kubernetes.io~projected/kube-api-access-782wd:{mountpoint:/var/lib/kubelet/pods/274f3045-f615-43d6-8d73-ebf9199ea724/volumes/kubernetes.io~projected/kube-api-access-782wd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/362a27d3-e34b-4d55-996c-ad51b6423b50/volumes/kubernetes.io~projected/kube-api-access-hvmn4:{mountpoint:/var/lib/kubelet/pods/362a27d3-e34b-4d55-996c-ad51b6423b50/volumes/kubernetes.io~projected/kube-api-access-hvmn4 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/450acb37-2b14-4264-b470-ce0a63835e8b/volumes/kubernetes.io~projected/kube-api-access-tlc8g:{mountpoint:/var/lib/kubelet/pods/450acb37-2b14-4264-b470-ce0a63835e8b/volumes/kubernetes.io~projected/kube-api-access-tlc8g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4657e9d9-df67-410c-8154-655ecd80a494/volumes/kubernetes.io~projected/kube-api-access-tf4v6:{mountpoint:/var/lib/kubelet/pods/4657e9d9-df67-410c-8154-655ecd80a494/volumes/kubernetes.io~projected/kube-api-access-tf4v6 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985/volumes/kubernetes.io~projected/kube-api-access-4rs7z:{mountpoint:/var/lib/kubelet/pods/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985/volumes/kubernetes.io~projected/kube-api-access-4rs7z major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/597ba6ab-ea33-4347-871a-f4edc6188d05/volumes/kubernetes.io~projected/kube-api-access-qrlkq:{mountpoint:/var/lib/kubelet/pods/597ba6ab-ea33-4347-871a-f4edc6188d05/volumes/kubernetes.io~projected/kube-api-access-qrlkq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f23597-a6bb-4e60-a54f-b7411e073675/volumes/kubernetes.io~projected/kube-api-access-kdcp9:{mountpoint:/var/lib/kubelet/pods/59f23597-a6bb-4e60-a54f-b7411e073675/volumes/kubernetes.io~projected/kube-api-access-kdcp9 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65ac9b7f-736c-405d-babf-089ef215fb8a/volumes/kubernetes.io~projected/kube-api-access-s64c4:{mountpoint:/var/lib/kubelet/pods/65ac9b7f-736c-405d-babf-089ef215fb8a/volumes/kubernetes.io~projected/kube-api-access-s64c4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66166d99-28a5-4a07-af27-2f605fb566df/volumes/kubernetes.io~projected/kube-api-access-dpxqb:{mountpoint:/var/lib/kubelet/pods/66166d99-28a5-4a07-af27-2f605fb566df/volumes/kubernetes.io~projected/kube-api-access-dpxqb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/661b34e8-d977-46ae-a3c6-6f09a1182f8d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/661b34e8-d977-46ae-a3c6-6f09a1182f8d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~projected/kube-api-access-9mlz7:{mountpoint:/var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~projected/kube-api-access-9mlz7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~projected/kube-api-access-cwzts:{mountpoint:/var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~projected/kube-api-access-cwzts major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1/volumes/kubernetes.io~projected/kube-api-access-5zq6l:{mountpoint:/var/lib/kubelet/pods/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1/volumes/kubernetes.io~projected/kube-api-access-5zq6l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87a5d53c-512b-48d9-be21-a6f82b3f3761/volumes/kubernetes.io~projected/kube-api-access-kdl4b:{mountpoint:/var/lib/kubelet/pods/87a5d53c-512b-48d9-be21-a6f82b3f3761/volumes/kubernetes.io~projected/kube-api-access-kdl4b major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~projected/kube-api-access-kfmbh:{mountpoint:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~projected/kube-api-access-kfmbh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~projected/kube-api-access-mzghf:{mountpoint:/var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~projected/kube-api-access-mzghf major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95598dfe-8d0c-4a37-9a8f-479bb69f07c0/volumes/kubernetes.io~projected/kube-api-access-s2mzh:{mountpoint:/var/lib/kubelet/pods/95598dfe-8d0c-4a37-9a8f-479bb69f07c0/volumes/kubernetes.io~projected/kube-api-access-s2mzh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097/volumes/kubernetes.io~projected/kube-api-access-rvlg6:{mountpoint:/var/lib/kubelet/pods/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097/volumes/kubernetes.io~projected/kube-api-access-rvlg6 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e9c510-1535-4f63-adc0-f4de450b6900/volumes/kubernetes.io~projected/kube-api-access-mhk27:{mountpoint:/var/lib/kubelet/pods/a6e9c510-1535-4f63-adc0-f4de450b6900/volumes/kubernetes.io~projected/kube-api-access-mhk27 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~projected/kube-api-access-7fxk2:{mountpoint:/var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~projected/kube-api-access-7fxk2 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/kube-api-access-zjvnn:{mountpoint:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/kube-api-access-zjvnn major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/kube-api-access-jr9fh:{mountpoint:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/kube-api-access-jr9fh major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cda96db7-532a-4209-8761-1ef2ba00260e/volumes/kubernetes.io~projected/kube-api-access-cnlnw:{mountpoint:/var/lib/kubelet/pods/cda96db7-532a-4209-8761-1ef2ba00260e/volumes/kubernetes.io~projected/kube-api-access-cnlnw major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf65cc24-9b6c-4723-903e-c0b0a320ac8b/volumes/kubernetes.io~projected/kube-api-access-cm4p7:{mountpoint:/var/lib/kubelet/pods/cf65cc24-9b6c-4723-903e-c0b0a320ac8b/volumes/kubernetes.io~projected/kube-api-access-cm4p7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0472439-1511-4476-8d44-ffcc69b07db8/volumes/kubernetes.io~projected/kube-api-access-ns2x8:{mountpoint:/var/lib/kubelet/pods/d0472439-1511-4476-8d44-ffcc69b07db8/volumes/kubernetes.io~projected/kube-api-access-ns2x8 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~projected/kube-api-access-zhwf4:{mountpoint:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~projected/kube-api-access-zhwf4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e57fcd21-a810-4e2f-880b-d99b80541c6a/volumes/kubernetes.io~projected/kube-api-access-b5rhg:{mountpoint:/var/lib/kubelet/pods/e57fcd21-a810-4e2f-880b-d99b80541c6a/volumes/kubernetes.io~projected/kube-api-access-b5rhg major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~projected/kube-api-access-qjmmv:{mountpoint:/var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~projected/kube-api-access-qjmmv major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eca1be8a-3b84-4ef6-9a53-28e189b9cae0/volumes/kubernetes.io~projected/kube-api-access-nl5jl:{mountpoint:/var/lib/kubelet/pods/eca1be8a-3b84-4ef6-9a53-28e189b9cae0/volumes/kubernetes.io~projected/kube-api-access-nl5jl major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d5d309-c79f-4f6d-bde8-d37d617eb61d/volumes/kubernetes.io~projected/kube-api-access-lwdjq:{mountpoint:/var/lib/kubelet/pods/f9d5d309-c79f-4f6d-bde8-d37d617eb61d/volumes/kubernetes.io~projected/kube-api-access-lwdjq major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a17e4709f12b65c61319092cd1233dba06ab93da1d3ca72b94091f09e5cd4c43/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/84800500b2cadd17f5f2f8292d868d5ca47fad63c8cd52feaff02dbefe410d1e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7073de4fd4bbc8396b377ffd6f436b4e130308ad10b3211997c35dc1d7d9cef2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8cce48ae558b5bfec7c8806b080cb044cce9222c0caaf4b97430eaa7392e9d82/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/49e1d85aed337802cf35df3642af6ad0b453ea081933a1c300dc3f1d6abc8f38/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/72a8fecfdaeb5c8b09748ca0194ec5b1fd42da1b7256f44119ef3b6164ffaf54/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/acc5ab6fbfce653ff52b0a7041f84f1db449364cad1b91c2a1f8bed4945428f5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/da6b6d463d914a346bef3a6e718952b7c8873f5525f60543e3c36c6e3d8b5df3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/a6d2e0dc2410eccde9bbb69eeb38ff828fae9e887b7c32fe2f4a121f8b5d6c73/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/b066854be0d9177041e357b58c1b62919fba4563d5306815a1619351fb6bb31e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/53dc07f0f0fec781165600241d97d36a6abe8ecc56cb739873a7a4fe4911c8d2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/def77d6e51b50df388e50da4108701480d8b7ccc00a53e0cefc064ce2cb3fe92/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9154ec5374dac564c0373f2ed6530eadd4334dbeb467ce2c8f8b452754342e56/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/794c890ca1a26f989c140c831e88b03ece6cca6a5e523d6714dfbab25ae4b206/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/181c7f60ba8560623512e432776451c6792624062f3b57429a2ecb5770861bab/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/68768dbae29167028e17402a3160b8bdd7f29151fcdf6068b35ef0b00ff2c3b8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/15d71b51d959e132628122b9dbb834eb6113f1f70f4f93c6789b90c62345dd37/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/49f0652dd87efdf694a33576425ae99dfd2b996e4ca73d6c914a62902a22b50d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f0d65d8bcd69c9b03bd66d7b56972a7a4df5d4c2ffff3a3f31b91d778256840f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/032a1bd35f5d1924c53bb8c44ae228c81e206b47ff94595625ae18d4087adf81/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4150f69cd5917572f128d9a71474c2af6bd7219a9ee024ab0858617acc797154/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/1bb6aa85ed4662bb798526c76b295e8cc8c7659f85298ad66f6c8feb255b8f7d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/26462f442d63ba920ae75b15bffed549cfab36b3e92a486ae97477772d588cff/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/dd8a310ad6a6cd827214a9ff5be5f2888e53c05118cd29f2e34c4955b1781b7c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/19d7b71fc6000dcac08ac2cfae2cc34f911bdbd01c7cd3aefe3de9a1265a9e95/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/47fa5f8e831f4cb0080c14d73bb1f8f7e71c8f9814746bac5f0d48f8dec40075/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/0aec8c21b7b39d4955ec2a74aa2c935150d661ed1a2759238eb2bf7c09225fbb/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e146d412ed47b4dc0f96c3e62d5cae4fa225a3b6f01df70a9e1e6798f4eaacbd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/00b5b6a327556f8cbf26acff900b84c33e280c2b8922f0803c34904b0a479f99/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/bc0cf20f1ba62be4375daba814b10e8d6a33c66167098bf22cba3e1a9080eb6c/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/0dd36c58afca35b1fc20741c8b5ac97a01efe9969aad659b743cab52ac2a5eec/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/37af5aac338e4569b8bad4e6650e011bca466f2f73006d9c686d769f13ed984b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/0b175fb16204f45ea8ba95103c79352ee671735630e60b6f8d3949020cbd23bb/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a577a477ae8a5d6a383c4745277f270381a77bc78801bac13fde8716963b7ee8/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/89b3ce1c28737ff9118a9608102fd982f9247afcf6971f2afcd61e4ecd77444f/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/db3554e0fbff2d84dedcddd6aa9cd4b522175cc4e914c3aab637893ec0bc7f77/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/f43bb822b728b8ebfb198cacbc52de2ad2c8286406e054ce89a5c4eb4e9b1a40/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/6816c076c0ab8f13af9f9b80edae3ff8d10759982b22ebe7593e97705ca0f3bc/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/68971ab444b8c5da3e0223f69d3458745ac544e464cb0cfdbb01963cecd6df3c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/4c000da349c240e03d7c6675c6103d88edcd380d397c6b1a0ed39afbcd9fe5c3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/9fd3ea0e05efd704caccd62d685209352e8ad57b27ad849ea7c9a8ab4e806969/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/93f9a34e1a849afbb0f1a71fc0707ba413b1f100a1421c675c4809ebbf85219a/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/80597640072cb3828dcd8b84e3eb5cec86c10ea332b836daaf28c41420ed5ba1/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/52b838944970a70e1235f6d602fc8d4c85a49fe9bb0fa76cd29c0fd137559f83/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/9430582b7632603648a3d9c894c864bce173703594d91e8610181c62b44bfb20/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/a7e34d423d8d9b017aa23add196e169bb59126b0049ebf70ce8721c3b6ba56ac/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/833c97fbf3762f781789c9171e456cae11af3e33aeda028e4dc505781c085144/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d04bac4426b57c48ebfdc6a879850aafc16b89d385bfd9712a661db30f7c1e2e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b83fcd97160cb00f0f58ffca2fb0d4b62a5d5e108235dc9ab32100a2e3038eb0/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/ea3e6cef6ddca2d5a9e5eacc052eacd30b03544cfccdc13e2bbba07c97004649/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-46:{mountpoint:/var/lib/containers/storage/overlay/111a5fa3b373e59ad487cb999d4e37fec2a909569951f097f3d639d0db4937d3/merged major:0 minor:46 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c7bb3108cba378a739750c621d103fa9961ca5b57a45e718fc9ce4624dfbbd15/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c47cbd2c699171dafc681bf4a971c7181c09a74aba288e334d2b1ad5f4051a4f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c40595cc0f1bf1ebd0ed2feba7625abbce669f20976724fef7eb8bf72160aaf3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/eb55f0975e9766e9b42ef3f7838d1149cb648a538797c46f7432b9eaf78aad27/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bd6f3f078b54383c2883086397a9e228e4d3816df78d348dcf3757e48ad15b89/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/96c5865820f01a671ca67a5860ad598c5b28addadc330e3ed2dbf93b07f63b45/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7ccd7574edb05c4f35ceda46598a27cc9860d62f7f87489e82593b3f8e4e5d35/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/db3792764b6cd4f979c6e94c98518543afc6d31eb8e72ec7994118c8132da2f2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/43effffd1862af9710c54bc1f694afaeffbc8b2d76ec994e9f199b42de0944ac/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8c8a545a80815d500cb656f35ed04aa9289db3690623d4b07a69958151909e16/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:22:06.343193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.342672 5051 manager.go:217] Machine: {Timestamp:2026-11-03 22:22:06.341724971 +0000 UTC m=+0.079069199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081402 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23f5dc7060e2436d16b1e1bf75ba5b SystemUUID:ec23f5dc-7060-e243-6d16-b1e1bf75ba5b BootID:937c0b06-d9f0-4dfe-9102-feb7054c86dc Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/66166d99-28a5-4a07-af27-2f605fb566df/volumes/kubernetes.io~projected/kube-api-access-dpxqb DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/802af844592653c3adc3e889b031b19992a9b727bc8c2b8d51cef728eab7b77c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~projected/kube-api-access-krz9j DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1dbf8b8e298a7455aa23de054957bbe6763461b585d1a4388f11f4818283235/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/986e59d71af668c327f17cd442300b94462044fadd32fa683824e114d4de5c03/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e57fcd21-a810-4e2f-880b-d99b80541c6a/volumes/kubernetes.io~projected/kube-api-access-b5rhg DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dc0d1af4dd44d4b2396a25e41093d71377a841d55e97c5cf46136f08df381e8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/450acb37-2b14-4264-b470-ce0a63835e8b/volumes/kubernetes.io~projected/kube-api-access-tlc8g DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e9c510-1535-4f63-adc0-f4de450b6900/volumes/kubernetes.io~projected/kube-api-access-mhk27 DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e906de24-2e38-4a47-81cd-581b7c617454/volumes/kubernetes.io~projected/kube-api-access-qjmmv DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78b4cc8663bec5194d416514a5151d2dce07ceda63b0cf35e0aeb98154cfe30e/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/796fb93d548cbf4d5a6c964ef420d5b064a0a4109175a06751fe17a3ddca5308/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e16b35d922f7ca72036089cc0ce2f7bd23451b2efb18f6063f4b327d98ff2e78/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/216f1639a8b643850f6470f0e6150e0080a5f0c15857f2ef5ed7ed6fcfe7265a/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5233dc64cd77bbcef484e54a9c9321f916e5c78f7758dd62086bef70dc25184/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4657e9d9-df67-410c-8154-655ecd80a494/volumes/kubernetes.io~projected/kube-api-access-tf4v6 DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d945ded0fa94f63c8ee76ce2a3595a8e5d9f121395a413551b786bf9afdd178f/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87a5d53c-512b-48d9-be21-a6f82b3f3761/volumes/kubernetes.io~projected/kube-api-access-kdl4b DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adc6656ccceaaa6a0725df9290ff81716e3c760954ec6fe8765497b8a98044b3/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0282c8b74c4c479cf9d87b7698416797d5a51d57029261949e580a46b410dd0/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cda96db7-532a-4209-8761-1ef2ba00260e/volumes/kubernetes.io~projected/kube-api-access-cnlnw DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59f23597-a6bb-4e60-a54f-b7411e073675/volumes/kubernetes.io~projected/kube-api-access-kdcp9 DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1cba54f6903c08acd2367f72c6c73935f4c296f6613e58fee76c04600400ddf/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b4e6a86c08e1d39eb96169894e5e1a3fd42581a4b6741c8ee47e24ee2d436ce/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097/volumes/kubernetes.io~projected/kube-api-access-rvlg6 DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ca5248b75c14d790d9160eb0d4888c9b5d136433ae58ca850d15f1939bec37f/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/661b34e8-d977-46ae-a3c6-6f09a1182f8d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985/volumes/kubernetes.io~projected/kube-api-access-4rs7z DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~projected/kube-api-access-zhwf4 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/kube-api-access-zjvnn DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95598dfe-8d0c-4a37-9a8f-479bb69f07c0/volumes/kubernetes.io~projected/kube-api-access-s2mzh DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/274f3045-f615-43d6-8d73-ebf9199ea724/volumes/kubernetes.io~projected/kube-api-access-782wd DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d5d309-c79f-4f6d-bde8-d37d617eb61d/volumes/kubernetes.io~projected/kube-api-access-lwdjq DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a73cbec62d0af633905226b9a39dfd6b1f0cf52c77f5147cbe16ddf495f2d11a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cf65cc24-9b6c-4723-903e-c0b0a320ac8b/volumes/kubernetes.io~projected/kube-api-access-cm4p7 DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/kube-api-access-jr9fh DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0472439-1511-4476-8d44-ffcc69b07db8/volumes/kubernetes.io~projected/kube-api-access-ns2x8 DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~projected/kube-api-access-9mlz7 DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cc9aafe61e33a888f86620caf8d88637f8e623404b198571b4cfcc957174a63/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae/volumes/kubernetes.io~projected/kube-api-access-mzghf DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~projected/kube-api-access-pj5jl DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-46 DeviceMajor:0 DeviceMinor:46 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~projected/kube-api-access-kfmbh DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ebf34f0bf961d7ea1bdb48a89f7b2171ce0dc1b5b9fc65aa046854179798295/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1/volumes/kubernetes.io~projected/kube-api-access-5zq6l DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17f1540d-7f49-4949-9679-4c6ca2ba9de9/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/volumes/kubernetes.io~projected/kube-api-access-cwzts DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/854c0c01c124266375a877256e3b8da898492a83e42df7d817a1662a61d55bf8/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae06052bca88794beab51918bdfa180947db1c8e461eaf7eaebb55fdee5df885/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f77da67-81d4-4a4e-a92b-c69542057b14/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30c152cb9d7ddbd2848b5a4e0266d4e06c45d8503af62670f94a7718a41cefd0/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/362a27d3-e34b-4d55-996c-ad51b6423b50/volumes/kubernetes.io~projected/kube-api-access-hvmn4 DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a8192fe-9bfe-4b5e-ab36-977ddf31533e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/volumes/kubernetes.io~projected/kube-api-access-7fxk2 DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6a01977-b5b7-4177-bfd4-e5d2265b442e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/65ac9b7f-736c-405d-babf-089ef215fb8a/volumes/kubernetes.io~projected/kube-api-access-s64c4 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5f500b742e75fc6801d69589ae3c3e2a61045fbd9d0d34154234e3c7496b6e2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eca1be8a-3b84-4ef6-9a53-28e189b9cae0/volumes/kubernetes.io~projected/kube-api-access-nl5jl DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/597ba6ab-ea33-4347-871a-f4edc6188d05/volumes/kubernetes.io~projected/kube-api-access-qrlkq DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8fb93ab2863fb1433f87038aac7b1d087ab87a0f2f9ab2b41cdfcd67112aa96/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d30d2f85dbdcaf86bdc1eab0f06463cb47399568401ba35d6e3ba17702e9cfd/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1dc0d1af4dd44d4 MacAddress:c6:33:f4:1c:54:74 Speed:10000 Mtu:8901} {Name:216f1639a8b6438 MacAddress:26:04:51:92:97:78 Speed:10000 Mtu:8901} {Name:4ca5248b75c14d7 MacAddress:36:3c:66:cc:4e:9d Speed:10000 Mtu:8901} {Name:6d30d2f85dbdcaf MacAddress:6a:57:e9:72:d4:f6 Speed:10000 Mtu:8901} {Name:78b4cc8663bec51 MacAddress:56:59:e1:f3:a1:8a Speed:10000 Mtu:8901} {Name:796fb93d548cbf4 MacAddress:46:05:28:9d:b3:9e Speed:10000 Mtu:8901} {Name:854c0c01c124266 MacAddress:a2:85:d9:94:3f:99 Speed:10000 Mtu:8901} {Name:986e59d71af668c MacAddress:ba:91:dd:f6:4e:4f Speed:10000 Mtu:8901} {Name:9b4e6a86c08e1d3 MacAddress:c6:bf:16:48:2b:59 Speed:10000 Mtu:8901} {Name:a0282c8b74c4c47 MacAddress:72:ea:a0:aa:45:cc Speed:10000 Mtu:8901} {Name:adc6656ccceaaa6 MacAddress:ee:02:f3:af:c7:e0 Speed:10000 Mtu:8901} {Name:b8fb93ab2863fb1 MacAddress:76:b5:f4:a9:1b:f1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:79:e8:6c:f8:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:6d:94:64:e4:15 Speed:0 Mtu:8901} {Name:c1cba54f6903c08 MacAddress:02:ac:f6:bd:34:76 Speed:10000 Mtu:8901} {Name:c1dbf8b8e298a74 MacAddress:c6:b7:3a:ef:18:30 Speed:10000 Mtu:8901} {Name:c5233dc64cd77bb MacAddress:5e:99:e5:45:18:56 Speed:10000 Mtu:8901} {Name:d945ded0fa94f63 MacAddress:fa:1a:ee:4d:98:57 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:79:e8:6c:f8:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1e:5a:6e:3f:62:eb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:d4:78:b3:75:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:22:06.343193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343189 5051 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:22:06.343532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343259 5051 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:22:06.343687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343644 5051 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:22:06.343875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343692 5051 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-87.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:22:06.343920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343880 5051 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:22:06.343920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343897 5051 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:22:06.343920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343912 5051 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:06.344021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.343934 5051 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:06.344074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344065 5051 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:22:06.344275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344263 5051 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:22:06.344388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344378 5051 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:22:06.344439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344420 5051 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:22:06.344439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344435 5051 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:22:06.344532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344446 5051 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:22:06.344532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344455 5051 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:22:06.344532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.344464 5051 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:22:06.345348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.345329 5051 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:06.345348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.345344 5051 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:06.345898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.345877 5051 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:22:06.349166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349150 5051 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:06.349166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349164 5051 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:06.349281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349271 5051 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:22:06.349482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349470 5051 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:22:06.349514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349500 5051 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:22:06.349617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349608 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:22:06.349645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349623 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:22:06.349645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349630 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:22:06.349645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349636 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:22:06.349645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349643 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349651 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349658 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349664 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349671 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349677 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349690 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349700 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349728 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:22:06.349756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.349737 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:22:06.353383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.353371 5051 server.go:1263] "Started kubelet" Nov 03 22:22:06.354909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.353444 5051 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:22:06.354909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.353469 5051 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:22:06.354909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.354309 5051 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:22:06.353954 ip-10-0-135-87 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:22:06.355235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.355146 5051 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:22:06.355420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.355340 5051 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:22:06.358146 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358127 5051 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358152 5051 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358193 5051 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:26 +0000 UTC" deadline="2028-05-29 01:01:28.334286459 +0000 UTC" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358222 5051 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358228 5051 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13730h39m21.976061687s" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358233 5051 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:22:06.358247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.358243 5051 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:22:06.359500 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.359352 5051 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:22:06.362198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362178 5051 factory.go:153] Registering CRI-O factory Nov 03 22:22:06.362305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362294 5051 factory.go:223] Registration of the crio container factory successfully Nov 03 22:22:06.362405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362385 5051 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:22:06.362405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362401 5051 factory.go:55] Registering systemd factory Nov 03 22:22:06.362577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362414 5051 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:22:06.362577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362455 5051 factory.go:103] Registering Raw factory Nov 03 22:22:06.362577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.362470 5051 manager.go:1199] Started watching for new ooms in manager Nov 03 22:22:06.366508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.366483 5051 manager.go:319] Starting recovery of all containers Nov 03 22:22:06.367395 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367352 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367398 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6a01977-b5b7-4177-bfd4-e5d2265b442e" volumeName="kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367411 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367425 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="362a27d3-e34b-4d55-996c-ad51b6423b50" volumeName="kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367439 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66166d99-28a5-4a07-af27-2f605fb566df" volumeName="kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367451 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae" volumeName="kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367464 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d6573e-1446-4ea2-b6f1-9cf9e19a161d" volumeName="kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367476 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle" seLinuxMountContext="" Nov 03 22:22:06.367494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367491 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a5d53c-512b-48d9-be21-a6f82b3f3761" volumeName="kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367503 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367516 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367528 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f1540d-7f49-4949-9679-4c6ca2ba9de9" volumeName="kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367541 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a5d53c-512b-48d9-be21-a6f82b3f3761" volumeName="kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367553 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8192fe-9bfe-4b5e-ab36-977ddf31533e" volumeName="kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367568 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57fcd21-a810-4e2f-880b-d99b80541c6a" volumeName="kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367582 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eca1be8a-3b84-4ef6-9a53-28e189b9cae0" volumeName="kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367613 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ac9b7f-736c-405d-babf-089ef215fb8a" volumeName="kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367626 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66166d99-28a5-4a07-af27-2f605fb566df" volumeName="kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367664 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8192fe-9bfe-4b5e-ab36-977ddf31533e" volumeName="kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367680 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367693 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72d9997c-a4f7-4f1a-8595-5a4e35c4ea65" volumeName="kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367706 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367720 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367731 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" volumeName="kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367756 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="597ba6ab-ea33-4347-871a-f4edc6188d05" volumeName="kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367769 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f23597-a6bb-4e60-a54f-b7411e073675" volumeName="kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367781 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f77da67-81d4-4a4e-a92b-c69542057b14" volumeName="kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367793 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d6573e-1446-4ea2-b6f1-9cf9e19a161d" volumeName="kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367809 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367822 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367835 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6a01977-b5b7-4177-bfd4-e5d2265b442e" volumeName="kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367847 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eca1be8a-3b84-4ef6-9a53-28e189b9cae0" volumeName="kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367861 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4657e9d9-df67-410c-8154-655ecd80a494" volumeName="kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367874 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="661b34e8-d977-46ae-a3c6-6f09a1182f8d" volumeName="kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs" seLinuxMountContext="" Nov 03 22:22:06.367878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367887 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95598dfe-8d0c-4a37-9a8f-479bb69f07c0" volumeName="kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367900 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf65cc24-9b6c-4723-903e-c0b0a320ac8b" volumeName="kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367913 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57fcd21-a810-4e2f-880b-d99b80541c6a" volumeName="kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367924 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" volumeName="kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367936 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985" volumeName="kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367955 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1" volumeName="kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.367972 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae" volumeName="kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368002 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae" volumeName="kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368015 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368028 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6a01977-b5b7-4177-bfd4-e5d2265b442e" volumeName="kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368041 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66166d99-28a5-4a07-af27-2f605fb566df" volumeName="kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368064 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a5d53c-512b-48d9-be21-a6f82b3f3761" volumeName="kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368078 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e9c510-1535-4f63-adc0-f4de450b6900" volumeName="kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368090 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368103 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368115 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f1540d-7f49-4949-9679-4c6ca2ba9de9" volumeName="kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368129 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8192fe-9bfe-4b5e-ab36-977ddf31533e" volumeName="kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368140 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368155 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf65cc24-9b6c-4723-903e-c0b0a320ac8b" volumeName="kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368165 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368177 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d5d309-c79f-4f6d-bde8-d37d617eb61d" volumeName="kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368188 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17f1540d-7f49-4949-9679-4c6ca2ba9de9" volumeName="kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368210 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450acb37-2b14-4264-b470-ce0a63835e8b" volumeName="kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368223 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f77da67-81d4-4a4e-a92b-c69542057b14" volumeName="kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368235 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368247 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf65cc24-9b6c-4723-903e-c0b0a320ac8b" volumeName="kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368261 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0472439-1511-4476-8d44-ffcc69b07db8" volumeName="kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368304 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0c6b7b5-cbbd-4486-a045-35348021f72e" volumeName="kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368318 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57fcd21-a810-4e2f-880b-d99b80541c6a" volumeName="kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368330 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="450acb37-2b14-4264-b470-ce0a63835e8b" volumeName="kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368344 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ac9b7f-736c-405d-babf-089ef215fb8a" volumeName="kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368355 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8192fe-9bfe-4b5e-ab36-977ddf31533e" volumeName="kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368368 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95598dfe-8d0c-4a37-9a8f-479bb69f07c0" volumeName="kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368381 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5ea7495-41a1-4f01-9b7e-c6ec6cf46097" volumeName="kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368393 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368405 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75e1082-c755-4119-9be9-328b7f4d91df" volumeName="kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368419 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda96db7-532a-4209-8761-1ef2ba00260e" volumeName="kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368445 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e9c510-1535-4f63-adc0-f4de450b6900" volumeName="kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368458 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d6573e-1446-4ea2-b6f1-9cf9e19a161d" volumeName="kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368470 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57fcd21-a810-4e2f-880b-d99b80541c6a" volumeName="kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368482 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e906de24-2e38-4a47-81cd-581b7c617454" volumeName="kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368495 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" volumeName="kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368507 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" volumeName="kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368522 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274f3045-f615-43d6-8d73-ebf9199ea724" volumeName="kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368548 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66166d99-28a5-4a07-af27-2f605fb566df" volumeName="kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368560 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f77da67-81d4-4a4e-a92b-c69542057b14" volumeName="kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368572 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb" volumeName="kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368584 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1" volumeName="kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368607 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" volumeName="kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368618 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="661b34e8-d977-46ae-a3c6-6f09a1182f8d" volumeName="kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368631 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72d9997c-a4f7-4f1a-8595-5a4e35c4ea65" volumeName="kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368681 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a8192fe-9bfe-4b5e-ab36-977ddf31533e" volumeName="kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368694 5051 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:22:06.368859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.368705 5051 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:22:06.381651 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.381617 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:06.382763 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.382739 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:06.397939 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.397873 5051 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:22:06.399321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.399295 5051 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:22:06.399431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.399328 5051 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:22:06.399431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.399354 5051 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:22:06.399431 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.399395 5051 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:22:06.425690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.425665 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:06.425872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.425696 5051 generic.go:356] "Generic (PLEG): container finished" podID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" exitCode=255 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441048 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="0f4fa8058118a5732c91f0cf3b5d107f63f8b5465d31c97b2bda8cf8532be0ab" exitCode=0 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441077 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="52764549c4d0cda16ff985d19fd42852c7a0c6140cc6e1896fbd8cb2db4de0af" exitCode=0 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441091 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="bb2b2b5d601a459e4ebc689d344761b2e1e701234f2d892229fbee4aabe71aae" exitCode=0 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441101 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="9a24ecd0534cb1bf73a80fc76ec0f40ba302d5ad0af2dda623afbe322898170d" exitCode=0 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441110 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="29f895f7f3795825260f9a794e597a200e4d2ed9845d596c241effb1402c6c33" exitCode=0 Nov 03 22:22:06.441125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.441118 5051 generic.go:356] "Generic (PLEG): container finished" podID="66166d99-28a5-4a07-af27-2f605fb566df" containerID="60435a2213f641dad2368f0e2e29b429af7b3af34f71219472eaccbcbcf2b518" exitCode=0 Nov 03 22:22:06.446342 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.446295 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1edcc8793fd6be957384c61df114fef" containerID="ff21a127c9f6c72e3ba05048dbb1a614a5d50123ed131f18469315252c6a5bb0" exitCode=0 Nov 03 22:22:06.459433 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.459408 5051 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:22:06.468096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.468072 5051 manager.go:324] Recovery completed Nov 03 22:22:06.489099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.489066 5051 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:06.490671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.490632 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:06.490806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.490711 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:06.490806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.490730 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:06.493643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493613 5051 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:22:06.493643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493633 5051 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:06.493768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493633 5051 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:22:06.493768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493738 5051 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:22:06.493899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493883 5051 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:22:06.493932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493900 5051 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:22:06.493932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493918 5051 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:22:06.493932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493924 5051 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:22:06.493932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493929 5051 policy_none.go:50] "Start" Nov 03 22:22:06.494045 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493938 5051 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:22:06.494045 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.493947 5051 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:22:06.497254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.497238 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:06.497328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.497268 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:06.497328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.497281 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:06.499701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.499683 5051 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:22:06.499743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.499710 5051 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:22:06.499743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.499720 5051 policy_none.go:44] "Start" Nov 03 22:22:06.499908 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.499892 5051 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 22:22:06.504319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.504290 5051 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:22:06.507038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507022 5051 manager.go:341] "Starting Device Plugin manager" Nov 03 22:22:06.507118 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.507055 5051 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:22:06.507118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507068 5051 server.go:87] "Starting device plugin registration server" Nov 03 22:22:06.507354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507344 5051 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:22:06.507407 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507355 5051 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:22:06.507471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507450 5051 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:22:06.507611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507566 5051 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:22:06.507710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507621 5051 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:22:06.507710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.507629 5051 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:22:06.508926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.508838 5051 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:22:06.50760611Z","UUID":"69853aea-7783-4041-bed8-e49ef0b1bb55","Handler":null,"Name":"","Endpoint":""} Nov 03 22:22:06.509109 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.509090 5051 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:22:06.509183 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.509139 5051 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-87.us-west-1.compute.internal\" not found" Nov 03 22:22:06.509700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.509489 5051 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:22:06.509700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.509524 5051 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:22:06.558678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.558645 5051 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:06.558885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.558775 5051 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:06.558885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.558793 5051 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:06.558955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.558939 5051 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:22:06.607882 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.607851 5051 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:06.609101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.609080 5051 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:06.609250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.609108 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:06.609250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.609140 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:06.609250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.609153 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-87.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:06.609250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.609196 5051 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.700492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.700370 5051 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal"] Nov 03 22:22:06.701075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701033 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"6400a77c4e4822387f7b4ea34fe62bb6","Type":"ContainerStarted","Data":"0c925a81512f4ab30658f93038eb69f9e6242bba2b7b33740061d5f4bb1af6cf"} Nov 03 22:22:06.701128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701085 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"6400a77c4e4822387f7b4ea34fe62bb6","Type":"ContainerStarted","Data":"5152fe6e7c102314e057ce298c7edebf9a7388d9f54e76f121418e6a841baa2f"} Nov 03 22:22:06.701198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701087 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.701279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701267 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.701324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701181 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerStarted","Data":"030087c99c1ff5c954c97b99fff6c9234c4adbf9542aedea096b93644b96054b"} Nov 03 22:22:06.701359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701337 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerDied","Data":"ff21a127c9f6c72e3ba05048dbb1a614a5d50123ed131f18469315252c6a5bb0"} Nov 03 22:22:06.701359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.701353 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" event={"ID":"b1edcc8793fd6be957384c61df114fef","Type":"ContainerStarted","Data":"429aba91ab676912dae5ce68b6cee6b6581ba5ad5d08076209aa32d4284bbfa1"} Nov 03 22:22:06.760290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.760254 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.760290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.760292 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.760502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.760313 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.764758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.764738 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:06.764838 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.764792 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.765086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.765070 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:06.765118 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:06.765108 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.771147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.771126 5051 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.771219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.771198 5051 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861233 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861290 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861343 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861352 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6400a77c4e4822387f7b4ea34fe62bb6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"6400a77c4e4822387f7b4ea34fe62bb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:06.861461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:06.861352 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1edcc8793fd6be957384c61df114fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" (UID: \"b1edcc8793fd6be957384c61df114fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:07.344672 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.344640 5051 apiserver.go:52] "Watching apiserver" Nov 03 22:22:07.461147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.461119 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:07.461305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.461212 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:07.497109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.497084 5051 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:22:07.497829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.497805 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-plm6l","openshift-cluster-node-tuning-operator/tuned-dcbzg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j","kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-88khg","kube-system/kas-connection-checker-685895c7c9-z4nhs","openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd","openshift-multus/multus-h85gn","openshift-network-console/networking-console-plugin-86c478bb88-x6mwc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx","openshift-console-operator/console-operator-7cd56b8dbc-t8ztc","openshift-image-registry/image-registry-84854fd764-cb9xd","openshift-ingress/router-default-79487cbf4c-klb6d","openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-4pjvv","kube-system/konnectivity-agent-mlhk9","openshift-ingress-canary/ingress-canary-slgc6","openshift-ovn-kubernetes/ovnkube-node-cdcgs","openshift-multus/multus-additional-cni-plugins-tzg8f","openshift-multus/network-metrics-daemon-m8g4v","openshift-image-registry/image-registry-57dd5bc9b9-hpc6k","kube-system/kas-connection-checker-685895c7c9-9dn28","openshift-dns/dns-default-xgvtf","openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h","kube-system/kas-connection-checker-685895c7c9-l9j49","openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv","openshift-dns/node-resolver-dw62d","openshift-network-diagnostics/network-check-target-wmt2m","openshift-service-ca/service-ca-5fffc97bdf-8s2f2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh","openshift-image-registry/node-ca-6hzqr","openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh","openshift-insights/periodic-gathering-lkvmc-bgpw2","kube-system/global-pull-secret-syncer-dchqn"] Nov 03 22:22:07.499060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.499041 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.500239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.500221 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.500639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.500626 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.501120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.501101 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.503093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.503075 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:07.504063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.504050 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:07.512614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.512597 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.513181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.513163 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.514385 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.514367 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.516397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.516380 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.521274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.521259 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:07.521349 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.521303 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:07.521728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.521715 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:07.521807 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.521751 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.us-west-1.compute.internal" Nov 03 22:22:07.550511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.550481 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.550511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.550487 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.551010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.550980 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:22:07.551010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.551001 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:22:07.551426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.551405 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:22:07.551551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.551541 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:22:07.551703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.551687 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.551844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.551831 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.552279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552267 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.552483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552465 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.552524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552488 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:22:07.552524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552499 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:22:07.552524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552515 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552593 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552592 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552652 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552673 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552755 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:22:07.552783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552775 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552842 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.552867 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553020 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553045 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553056 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553090 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553184 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553241 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.553450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553340 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:22:07.553828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553477 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.553828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553665 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:22:07.553828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553810 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:22:07.553910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553840 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.553977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.553964 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:22:07.560828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.560810 5051 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:22:07.564203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564187 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.564266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564211 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.564266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564227 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.564266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.564266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564263 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.564397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564369 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-tmp\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.564397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564375 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.564524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564411 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.564524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564451 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.564524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-tmp-dir\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564551 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlz7\" (UniqueName: \"kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlkq\" (UniqueName: \"kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564607 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564630 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564656 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564685 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564697 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.564728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564710 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564752 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564805 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f77da67-81d4-4a4e-a92b-c69542057b14-config\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564815 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmn4\" (UniqueName: \"kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4\") pod \"migrator-66df5d496b-ggkcd\" (UID: \"362a27d3-e34b-4d55-996c-ad51b6423b50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564839 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564876 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564883 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.564918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565005 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565042 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565071 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmmv\" (UniqueName: \"kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565084 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.565125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565104 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdjq\" (UniqueName: \"kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565158 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565196 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565266 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565285 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565353 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565379 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565395 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdcp9\" (UniqueName: \"kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9\") pod \"kas-connection-checker-685895c7c9-l9j49\" (UID: \"59f23597-a6bb-4e60-a54f-b7411e073675\") " pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565428 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.565547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565585 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64c4\" (UniqueName: \"kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-stats-auth\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565616 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565656 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565661 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bbwr5\"" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565678 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565681 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565700 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565737 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565752 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565773 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565815 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565865 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565885 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565896 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-tmp-dir\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565904 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.565966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.565956 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566012 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566030 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566064 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566102 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566149 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566185 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566215 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566232 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566305 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlg6\" (UniqueName: \"kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6\") pod \"migrator-66df5d496b-8t2kh\" (UID: \"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566376 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566386 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzghf\" (UniqueName: \"kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566398 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566491 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566511 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566532 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566565 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566609 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzts\" (UniqueName: \"kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:22:07.566668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566671 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlc8g\" (UniqueName: \"kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566686 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566704 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566736 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566747 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566788 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566870 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566887 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566907 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566926 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.566969 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567017 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567038 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2x8\" (UniqueName: \"kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8\") pod \"network-check-source-6f78d59f99-vcq5h\" (UID: \"d0472439-1511-4476-8d44-ffcc69b07db8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567066 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567068 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-data-volume\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567081 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567106 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567112 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567123 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567138 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567154 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567181 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567211 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782wd\" (UniqueName: \"kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd\") pod \"volume-data-source-validator-58c9f5f46d-rsrsh\" (UID: \"274f3045-f615-43d6-8d73-ebf9199ea724\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567250 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-tuned\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567337 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567341 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-snapshots\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.567389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567381 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567413 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567460 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4p7\" (UniqueName: \"kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567481 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567517 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567516 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567559 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567588 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567606 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxk2\" (UniqueName: \"kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567630 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567684 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567697 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-default-certificate\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567710 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567736 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567764 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567789 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567882 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567900 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq6l\" (UniqueName: \"kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567926 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567941 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567945 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f77da67-81d4-4a4e-a92b-c69542057b14-serving-cert\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567955 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.567975 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568033 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568060 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568100 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mzh\" (UniqueName: \"kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568134 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krz9j\" (UniqueName: \"kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568173 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568203 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4v6\" (UniqueName: \"kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6\") pod \"kas-connection-checker-685895c7c9-z4nhs\" (UID: \"4657e9d9-df67-410c-8154-655ecd80a494\") " pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568232 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.568370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568259 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwf4\" (UniqueName: \"kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568407 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5jl\" (UniqueName: \"kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568432 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568455 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568471 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568534 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdl4b\" (UniqueName: \"kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568557 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568577 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568608 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568650 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rs7z\" (UniqueName: \"kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z\") pod \"kas-connection-checker-685895c7c9-9dn28\" (UID: \"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985\") " pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568675 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568703 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568753 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568774 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568792 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568814 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5jl\" (UniqueName: \"kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568844 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568870 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568897 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568924 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.568950 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569085 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmbh\" (UniqueName: \"kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569124 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569160 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxqb\" (UniqueName: \"kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569210 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.569399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569250 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569478 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569524 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569551 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569637 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569656 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569674 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569681 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569691 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhk27\" (UniqueName: \"kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569734 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569746 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/450acb37-2b14-4264-b470-ce0a63835e8b-serviceca\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569753 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e906de24-2e38-4a47-81cd-581b7c617454-tmp\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569755 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569793 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569825 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.570261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.569891 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.570935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.570919 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ft6gs\"" Nov 03 22:22:07.570980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.570934 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:22:07.570980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.570919 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:22:07.570980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.570934 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mx976\"" Nov 03 22:22:07.575505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.575488 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e906de24-2e38-4a47-81cd-581b7c617454-serving-cert\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.577796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.577780 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovn-node-metrics-cert\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.577892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.577875 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/661b34e8-d977-46ae-a3c6-6f09a1182f8d-agent-certs\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:22:07.599850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.599789 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-px8pz\"" Nov 03 22:22:07.599850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.599841 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2scz\"" Nov 03 22:22:07.600874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.600622 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:07.600874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.600700 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglhz\"" Nov 03 22:22:07.600874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.600788 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:22:07.601670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.601649 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2wh6\"" Nov 03 22:22:07.601960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.601945 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhswt\"" Nov 03 22:22:07.602114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602079 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:22:07.602180 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602170 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.602240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602223 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:22:07.602285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602240 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602327 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602403 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602517 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602683 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602769 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602853 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.602858 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25tw\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603023 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:22:07.603295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603026 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:22:07.603717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603312 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:22:07.603717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603389 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.603717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603429 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603571 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.603717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603686 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603740 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603813 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603826 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dx5sz\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603848 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603873 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.603922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603919 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.603970 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.604058 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.604066 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.604073 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.604107 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnsfs\"" Nov 03 22:22:07.604217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.604173 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.605172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.605154 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-serving-cert\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.605344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.605329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-original-pull-secret\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.606050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606032 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.606128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606031 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:22:07.606128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606117 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-env-overrides\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.606195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606137 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:22:07.606339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606325 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.606374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606325 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.606885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606867 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-config\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.606943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.606903 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-config\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.607052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.607037 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e0c6b7b5-cbbd-4486-a045-35348021f72e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:07.608393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.608376 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/661b34e8-d977-46ae-a3c6-6f09a1182f8d-konnectivity-ca\") pod \"konnectivity-agent-mlhk9\" (UID: \"661b34e8-d977-46ae-a3c6-6f09a1182f8d\") " pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:22:07.608444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.608381 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-ovnkube-script-lib\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.608444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.608395 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-daemon-config\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.608444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.608421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65ac9b7f-736c-405d-babf-089ef215fb8a-iptables-alerter-script\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.608533 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.608514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.609579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.609565 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906de24-2e38-4a47-81cd-581b7c617454-service-ca-bundle\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.609632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.609594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87a5d53c-512b-48d9-be21-a6f82b3f3761-cni-binary-copy\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.609728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.609711 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66166d99-28a5-4a07-af27-2f605fb566df-cni-binary-copy\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.615258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.615237 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.615330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.615237 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-trusted-ca\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.626529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.626500 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdjq\" (UniqueName: \"kubernetes.io/projected/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-kube-api-access-lwdjq\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:07.626938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.626921 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmmv\" (UniqueName: \"kubernetes.io/projected/e906de24-2e38-4a47-81cd-581b7c617454-kube-api-access-qjmmv\") pod \"insights-operator-7c4986d449-plm6l\" (UID: \"e906de24-2e38-4a47-81cd-581b7c617454\") " pod="openshift-insights/insights-operator-7c4986d449-plm6l" Nov 03 22:22:07.626975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.626920 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdcp9\" (UniqueName: \"kubernetes.io/projected/59f23597-a6bb-4e60-a54f-b7411e073675-kube-api-access-kdcp9\") pod \"kas-connection-checker-685895c7c9-l9j49\" (UID: \"59f23597-a6bb-4e60-a54f-b7411e073675\") " pod="kube-system/kas-connection-checker-685895c7c9-l9j49" Nov 03 22:22:07.629948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.629928 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64c4\" (UniqueName: \"kubernetes.io/projected/65ac9b7f-736c-405d-babf-089ef215fb8a-kube-api-access-s64c4\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.630041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.629937 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlz7\" (UniqueName: \"kubernetes.io/projected/6f77da67-81d4-4a4e-a92b-c69542057b14-kube-api-access-9mlz7\") pod \"service-ca-operator-544d749c8c-ckj7j\" (UID: \"6f77da67-81d4-4a4e-a92b-c69542057b14\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" Nov 03 22:22:07.632260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.632240 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.632335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.632319 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlnw\" (UniqueName: \"kubernetes.io/projected/cda96db7-532a-4209-8761-1ef2ba00260e-kube-api-access-cnlnw\") pod \"network-check-target-wmt2m\" (UID: \"cda96db7-532a-4209-8761-1ef2ba00260e\") " pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:22:07.635619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.635599 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.635708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.635688 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxqb\" (UniqueName: \"kubernetes.io/projected/66166d99-28a5-4a07-af27-2f605fb566df-kube-api-access-dpxqb\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.635768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.635755 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5jl\" (UniqueName: \"kubernetes.io/projected/17f1540d-7f49-4949-9679-4c6ca2ba9de9-kube-api-access-pj5jl\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.637207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.637185 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782wd\" (UniqueName: \"kubernetes.io/projected/274f3045-f615-43d6-8d73-ebf9199ea724-kube-api-access-782wd\") pod \"volume-data-source-validator-58c9f5f46d-rsrsh\" (UID: \"274f3045-f615-43d6-8d73-ebf9199ea724\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rsrsh" Nov 03 22:22:07.637309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.637296 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5jl\" (UniqueName: \"kubernetes.io/projected/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-api-access-nl5jl\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.637341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.637314 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmbh\" (UniqueName: \"kubernetes.io/projected/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-kube-api-access-kfmbh\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.638372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.638356 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlc8g\" (UniqueName: \"kubernetes.io/projected/450acb37-2b14-4264-b470-ce0a63835e8b-kube-api-access-tlc8g\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.638609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.638594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"periodic-gathering-lkvmc-bgpw2\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:22:07.641056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.641029 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mzh\" (UniqueName: \"kubernetes.io/projected/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-kube-api-access-s2mzh\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.641126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.641054 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krz9j\" (UniqueName: \"kubernetes.io/projected/0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3-kube-api-access-krz9j\") pod \"console-operator-7cd56b8dbc-t8ztc\" (UID: \"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:07.642332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.642309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxk2\" (UniqueName: \"kubernetes.io/projected/b5d6573e-1446-4ea2-b6f1-9cf9e19a161d-kube-api-access-7fxk2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6btx\" (UID: \"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" Nov 03 22:22:07.642459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.642443 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdl4b\" (UniqueName: \"kubernetes.io/projected/87a5d53c-512b-48d9-be21-a6f82b3f3761-kube-api-access-kdl4b\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.642562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.642514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.642744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.642726 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4v6\" (UniqueName: \"kubernetes.io/projected/4657e9d9-df67-410c-8154-655ecd80a494-kube-api-access-tf4v6\") pod \"kas-connection-checker-685895c7c9-z4nhs\" (UID: \"4657e9d9-df67-410c-8154-655ecd80a494\") " pod="kube-system/kas-connection-checker-685895c7c9-z4nhs" Nov 03 22:22:07.643760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.643738 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rs7z\" (UniqueName: \"kubernetes.io/projected/4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985-kube-api-access-4rs7z\") pod \"kas-connection-checker-685895c7c9-9dn28\" (UID: \"4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985\") " pod="kube-system/kas-connection-checker-685895c7c9-9dn28" Nov 03 22:22:07.643821 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.643770 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.645682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.645662 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zq6l\" (UniqueName: \"kubernetes.io/projected/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-kube-api-access-5zq6l\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.645803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.645784 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4p7\" (UniqueName: \"kubernetes.io/projected/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-kube-api-access-cm4p7\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.645974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.645961 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhk27\" (UniqueName: \"kubernetes.io/projected/a6e9c510-1535-4f63-adc0-f4de450b6900-kube-api-access-mhk27\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.650708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2x8\" (UniqueName: \"kubernetes.io/projected/d0472439-1511-4476-8d44-ffcc69b07db8-kube-api-access-ns2x8\") pod \"network-check-source-6f78d59f99-vcq5h\" (UID: \"d0472439-1511-4476-8d44-ffcc69b07db8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vcq5h" Nov 03 22:22:07.650799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650785 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:22:07.650853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650840 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:07.650899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650885 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650885 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650903 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650902 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650927 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650933 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650909 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650932 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:22:07.650947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650939 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.651340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.650972 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:22:07.651340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651127 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:07.651340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651284 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:22:07.651485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651472 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-697vs\"" Nov 03 22:22:07.651535 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651527 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cbh97\"" Nov 03 22:22:07.651582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651547 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:22:07.651582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651560 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rrf2w\"" Nov 03 22:22:07.651671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651623 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dbbtb\"" Nov 03 22:22:07.651671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651664 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651664 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vc5c5\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651685 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651689 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9kg8s\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651673 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q82l2\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651703 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-94lr2\"" Nov 03 22:22:07.651754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.651743 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-blfxv\"" Nov 03 22:22:07.653360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.653339 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlg6\" (UniqueName: \"kubernetes.io/projected/a5ea7495-41a1-4f01-9b7e-c6ec6cf46097-kube-api-access-rvlg6\") pod \"migrator-66df5d496b-8t2kh\" (UID: \"a5ea7495-41a1-4f01-9b7e-c6ec6cf46097\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t2kh" Nov 03 22:22:07.653499 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.653485 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzghf\" (UniqueName: \"kubernetes.io/projected/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-kube-api-access-mzghf\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.655401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.655386 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwf4\" (UniqueName: \"kubernetes.io/projected/d6a01977-b5b7-4177-bfd4-e5d2265b442e-kube-api-access-zhwf4\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.656341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.656320 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-key\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.656417 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.656400 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-cert\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:22:07.656595 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.656578 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlkq\" (UniqueName: \"kubernetes.io/projected/597ba6ab-ea33-4347-871a-f4edc6188d05-kube-api-access-qrlkq\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.656647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.656594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzts\" (UniqueName: \"kubernetes.io/projected/72d9997c-a4f7-4f1a-8595-5a4e35c4ea65-kube-api-access-cwzts\") pod \"ingress-canary-slgc6\" (UID: \"72d9997c-a4f7-4f1a-8595-5a4e35c4ea65\") " pod="openshift-ingress-canary/ingress-canary-slgc6" Nov 03 22:22:07.656897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.656879 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-config-volume\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.658143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.658129 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmn4\" (UniqueName: \"kubernetes.io/projected/362a27d3-e34b-4d55-996c-ad51b6423b50-kube-api-access-hvmn4\") pod \"migrator-66df5d496b-ggkcd\" (UID: \"362a27d3-e34b-4d55-996c-ad51b6423b50\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ggkcd" Nov 03 22:22:07.660142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.660128 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae-signing-cabundle\") pod \"service-ca-5fffc97bdf-8s2f2\" (UID: \"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.670689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670668 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.670746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670694 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.670746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670711 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.670746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670725 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.670746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670755 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670771 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670795 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670805 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-multus-certs\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670819 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670824 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ac9b7f-736c-405d-babf-089ef215fb8a-host-slash\") pod \"iptables-alerter-4pjvv\" (UID: \"65ac9b7f-736c-405d-babf-089ef215fb8a\") " pod="openshift-network-operator/iptables-alerter-4pjvv" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-bin\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670854 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-modprobe-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670874 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-system-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670806 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-sys-fs\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670899 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysconfig\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.670918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670906 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.670908 5051 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670937 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-log-socket\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670932 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670948 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-run\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670973 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1-hosts-file\") pod \"node-resolver-dw62d\" (UID: \"8595e2d4-2ee4-41e4-8844-eae4e7ecfdb1\") " pod="openshift-dns/node-resolver-dw62d" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.670909 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.670982 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671015 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671029 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171011595 +0000 UTC m=+1.908355820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671044 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671048 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-k8s-cni-cncf-io\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671064 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671081 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171066752 +0000 UTC m=+1.908410981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671098 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671103 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-crio-socket\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671117 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671136 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671151 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671199 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671199 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-ovn\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671207 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-var-lib-kubelet\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671222 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671246 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-cni-netd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671263 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-cni-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671297 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671304 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-systemd-units\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-dbus\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671317 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671275 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671310 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-systemd\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671346 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671356 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.671370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671365 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671394 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171378696 +0000 UTC m=+1.908722901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671414 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671411 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-node-log\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671443 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-socket-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671448 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671465 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671499 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-etc-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671551 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671566 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671593 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-lib-modules\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-multus\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-host\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671627 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671675 5051 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671714 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171705515 +0000 UTC m=+1.909049740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671726 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671742 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671764 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671780 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-cni-bin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671797 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171781717 +0000 UTC m=+1.909125940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671780 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671812 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-var-lib-kubelet\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-kubernetes\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671838 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-device-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671871 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671879 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671888 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671901 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.171894149 +0000 UTC m=+1.909238390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : secret "metrics-daemon-secret" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671914 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-conf-dir\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671916 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671935 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671951 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.671956 5051 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671938 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-host-run-netns\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671974 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.671969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-registration-dir\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672009 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/450acb37-2b14-4264-b470-ce0a63835e8b-host\") pod \"node-ca-6hzqr\" (UID: \"450acb37-2b14-4264-b470-ce0a63835e8b\") " pod="openshift-image-registry/node-ca-6hzqr" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672026 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.172010278 +0000 UTC m=+1.909354496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672054 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672075 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-etc-kubernetes\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672100 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672109 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-conf\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672124 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-system-cni-dir\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672129 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672167 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672166 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/597ba6ab-ea33-4347-871a-f4edc6188d05-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8r2j\" (UID: \"597ba6ab-ea33-4347-871a-f4edc6188d05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8r2j" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672185 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672208 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672219 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-kubelet\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672222 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-var-lib-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672248 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb-kubelet-config\") pod \"global-pull-secret-syncer-dchqn\" (UID: \"71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb\") " pod="kube-system/global-pull-secret-syncer-dchqn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672233 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672266 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-cnibin\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672295 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-multus-socket-dir-parent\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672295 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672314 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672346 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.172333708 +0000 UTC m=+1.909677911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672349 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66166d99-28a5-4a07-af27-2f605fb566df-cnibin\") pod \"multus-additional-cni-plugins-tzg8f\" (UID: \"66166d99-28a5-4a07-af27-2f605fb566df\") " pod="openshift-multus/multus-additional-cni-plugins-tzg8f" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672315 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-slash\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672368 5051 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672371 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672389 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672394 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.172385439 +0000 UTC m=+1.909729642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672400 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-run-openvswitch\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672413 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-systemd\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672429 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672445 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672461 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672469 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672458 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672489 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672460 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672513 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8192fe-9bfe-4b5e-ab36-977ddf31533e-host-run-netns\") pod \"ovnkube-node-cdcgs\" (UID: \"8a8192fe-9bfe-4b5e-ab36-977ddf31533e\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672513 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.172486319 +0000 UTC m=+1.909830522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672532 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87a5d53c-512b-48d9-be21-a6f82b3f3761-hostroot\") pod \"multus-h85gn\" (UID: \"87a5d53c-512b-48d9-be21-a6f82b3f3761\") " pod="openshift-multus/multus-h85gn" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-etc-sysctl-d\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672563 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672593 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.672618 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a01977-b5b7-4177-bfd4-e5d2265b442e-sys\") pod \"tuned-dcbzg\" (UID: \"d6a01977-b5b7-4177-bfd4-e5d2265b442e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcbzg" Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672627 5051 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:07.672740 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:07.672679 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:08.172665949 +0000 UTC m=+1.910010166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:07.802440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.802408 5051 scope.go:122] "RemoveContainer" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" Nov 03 22:22:07.816904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.816807 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" Nov 03 22:22:07.819032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.819012 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:22:07.994933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:07.994908 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-slgc6_72d9997c-a4f7-4f1a-8595-5a4e35c4ea65/serve-healthcheck-canary/0.log" Nov 03 22:22:08.177230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177133 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:08.177230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177186 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:08.177230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177214 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177250 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177287 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177344 5051 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177289 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177362 5051 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177375 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177346 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177331366 +0000 UTC m=+2.914675582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177395 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177401 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177409 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177412 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.17739866 +0000 UTC m=+2.914742875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : configmap "kube-rbac-proxy" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177458 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177427468 +0000 UTC m=+2.914771675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177482 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177472238 +0000 UTC m=+2.914816446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:08.177521 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177514 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177506553 +0000 UTC m=+2.914850778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177533 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177577 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177631 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177660 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177635 5051 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177693 5051 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177699 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177683 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.177727 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177738 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177723432 +0000 UTC m=+2.915067648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177760 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177747294 +0000 UTC m=+2.915091503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177777 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177766342 +0000 UTC m=+2.915110553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177783 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177791 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177783457 +0000 UTC m=+2.915127665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177804 5051 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177814 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177804272 +0000 UTC m=+2.915148508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : secret "metrics-daemon-secret" not found Nov 03 22:22:08.178057 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.177850 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:09.177840939 +0000 UTC m=+2.915185142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:22:08.193104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.193081 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8t2kh_a5ea7495-41a1-4f01-9b7e-c6ec6cf46097/migrator/0.log" Nov 03 22:22:08.392816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.392785 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ggkcd_362a27d3-e34b-4d55-996c-ad51b6423b50/migrator/0.log" Nov 03 22:22:08.465712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.465639 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/2.log" Nov 03 22:22:08.466087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.466072 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/1.log" Nov 03 22:22:08.466145 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.466104 5051 generic.go:356] "Generic (PLEG): container finished" podID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" exitCode=255 Nov 03 22:22:08.466199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.466176 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerDied","Data":"968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf"} Nov 03 22:22:08.466244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.466231 5051 scope.go:122] "RemoveContainer" containerID="11806753ef9d6853a67cf17d978b3f5c9a698853283dac6d7eed453558f6931a" Nov 03 22:22:08.466546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.466526 5051 scope.go:122] "RemoveContainer" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" Nov 03 22:22:08.466783 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:08.466760 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:08.467940 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.467908 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" event={"ID":"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae","Type":"ContainerStarted","Data":"b7c47afbc33758e7989cdd4852cc653da37e7489030367be558c6d72ced6a8f5"} Nov 03 22:22:08.468052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.467959 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8s2f2" event={"ID":"8cc4226e-0a8a-4fe1-8dee-6fd3aae61dae","Type":"ContainerStarted","Data":"d945ded0fa94f63c8ee76ce2a3595a8e5d9f121395a413551b786bf9afdd178f"} Nov 03 22:22:08.593729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:08.593697 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-g6btx_b5d6573e-1446-4ea2-b6f1-9cf9e19a161d/kube-storage-version-migrator-operator/0.log" Nov 03 22:22:09.187155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187123 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:09.187155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187218 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187246 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187270 5051 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187303 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187354 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187332753 +0000 UTC m=+4.924676974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187353 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187374 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187389 5051 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187423 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.18740795 +0000 UTC m=+4.924752153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187441 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls podName:95598dfe-8d0c-4a37-9a8f-479bb69f07c0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187430159 +0000 UTC m=+4.924774366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jdkv" (UID: "95598dfe-8d0c-4a37-9a8f-479bb69f07c0") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187480 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:09.187504 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187498 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187525 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187545 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187534637 +0000 UTC m=+4.924878861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187585 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187590 5051 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187628 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls podName:a6e9c510-1535-4f63-adc0-f4de450b6900 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187616428 +0000 UTC m=+4.924960645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8mrdt" (UID: "a6e9c510-1535-4f63-adc0-f4de450b6900") : secret "samples-operator-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187647 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187639987 +0000 UTC m=+4.924984201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187661 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187655184 +0000 UTC m=+4.924999387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187694 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187718 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187753 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187819 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187877 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs podName:f9d5d309-c79f-4f6d-bde8-d37d617eb61d nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187863561 +0000 UTC m=+4.925207786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs") pod "network-metrics-daemon-m8g4v" (UID: "f9d5d309-c79f-4f6d-bde8-d37d617eb61d") : secret "metrics-daemon-secret" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187837 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187916 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls podName:eca1be8a-3b84-4ef6-9a53-28e189b9cae0 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187905036 +0000 UTC m=+4.925249239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88khg" (UID: "eca1be8a-3b84-4ef6-9a53-28e189b9cae0") : secret "insights-runtime-extractor-tls" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.187875 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187879 5051 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:09.188122 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:09.187984 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:11.187973633 +0000 UTC m=+4.925317836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:22:09.474057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:09.473971 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/2.log" Nov 03 22:22:11.207524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207486 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207534 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207566 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207622 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207655 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207671 5051 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207680 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207718 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.207743 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207768 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls podName:cf65cc24-9b6c-4723-903e-c0b0a320ac8b nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.207747933 +0000 UTC m=+8.945092140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls") pod "dns-default-xgvtf" (UID: "cf65cc24-9b6c-4723-903e-c0b0a320ac8b") : secret "dns-default-metrics-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207676 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207790 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207794 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207809 5051 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207676 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207857 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207872 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.207854059 +0000 UTC m=+8.945198265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : configmap references non-existent config key: service-ca.crt Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207892 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.20788202 +0000 UTC m=+8.945226225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207908 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.207900332 +0000 UTC m=+8.945244534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207923 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs podName:17f1540d-7f49-4949-9679-4c6ca2ba9de9 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.207915803 +0000 UTC m=+8.945260006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs") pod "router-default-79487cbf4c-klb6d" (UID: "17f1540d-7f49-4949-9679-4c6ca2ba9de9") : secret "router-metrics-certs-default" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:11.207942 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:15.207933491 +0000 UTC m=+8.945277700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.208009 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:11.208089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.208092 5051 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:22:11.210667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.210644 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95598dfe-8d0c-4a37-9a8f-479bb69f07c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jdkv\" (UID: \"95598dfe-8d0c-4a37-9a8f-479bb69f07c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:11.210851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.210832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eca1be8a-3b84-4ef6-9a53-28e189b9cae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88khg\" (UID: \"eca1be8a-3b84-4ef6-9a53-28e189b9cae0\") " pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:11.210894 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.210581 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d5d309-c79f-4f6d-bde8-d37d617eb61d-metrics-certs\") pod \"network-metrics-daemon-m8g4v\" (UID: \"f9d5d309-c79f-4f6d-bde8-d37d617eb61d\") " pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:11.211059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.211042 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a6e9c510-1535-4f63-adc0-f4de450b6900-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8mrdt\" (UID: \"a6e9c510-1535-4f63-adc0-f4de450b6900\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:11.399485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.399449 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" Nov 03 22:22:11.400551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.400518 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" Nov 03 22:22:11.404820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.404798 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8g4v" Nov 03 22:22:11.415297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.415270 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88khg" Nov 03 22:22:11.553225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.552532 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:11.800728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.800641 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt"] Nov 03 22:22:11.850309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.850278 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv"] Nov 03 22:22:11.854350 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:11.854320 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95598dfe_8d0c_4a37_9a8f_479bb69f07c0.slice/crio-8ac584b62d0217d8a996a5c79c26cdcd428bb7825b1af2a7911456433613028d WatchSource:0}: Error finding container 8ac584b62d0217d8a996a5c79c26cdcd428bb7825b1af2a7911456433613028d: Status 404 returned error can't find the container with id 8ac584b62d0217d8a996a5c79c26cdcd428bb7825b1af2a7911456433613028d Nov 03 22:22:11.856181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.856155 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:22:11.903016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.902972 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88khg"] Nov 03 22:22:11.906826 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:11.906802 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca1be8a_3b84_4ef6_9a53_28e189b9cae0.slice/crio-b5b3b2984bc9c3292cb18b5bee2cd611539ae84d552b7a66bb680488b330acaf WatchSource:0}: Error finding container b5b3b2984bc9c3292cb18b5bee2cd611539ae84d552b7a66bb680488b330acaf: Status 404 returned error can't find the container with id b5b3b2984bc9c3292cb18b5bee2cd611539ae84d552b7a66bb680488b330acaf Nov 03 22:22:11.924950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:11.924925 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8g4v"] Nov 03 22:22:11.934163 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:11.934138 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d5d309_c79f_4f6d_bde8_d37d617eb61d.slice/crio-0826e04f5960c6836f5ae358a878b98cd1c967534e125546275fceb71369812e WatchSource:0}: Error finding container 0826e04f5960c6836f5ae358a878b98cd1c967534e125546275fceb71369812e: Status 404 returned error can't find the container with id 0826e04f5960c6836f5ae358a878b98cd1c967534e125546275fceb71369812e Nov 03 22:22:12.493015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:12.492959 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" event={"ID":"95598dfe-8d0c-4a37-9a8f-479bb69f07c0","Type":"ContainerStarted","Data":"8ac584b62d0217d8a996a5c79c26cdcd428bb7825b1af2a7911456433613028d"} Nov 03 22:22:12.494349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:12.494318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" event={"ID":"a6e9c510-1535-4f63-adc0-f4de450b6900","Type":"ContainerStarted","Data":"d5ef9e2bea5a81f41755cde82876f0c01ce3f6daa37f3fc4b8d1daab21847da2"} Nov 03 22:22:12.495525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:12.495502 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8g4v" event={"ID":"f9d5d309-c79f-4f6d-bde8-d37d617eb61d","Type":"ContainerStarted","Data":"0826e04f5960c6836f5ae358a878b98cd1c967534e125546275fceb71369812e"} Nov 03 22:22:12.497067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:12.497048 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88khg" event={"ID":"eca1be8a-3b84-4ef6-9a53-28e189b9cae0","Type":"ContainerStarted","Data":"4eaf285b03762a3e85ba8060a6d2d1feaa60d8d23fd0b25166333c1b890e3e0e"} Nov 03 22:22:12.497167 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:12.497088 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88khg" event={"ID":"eca1be8a-3b84-4ef6-9a53-28e189b9cae0","Type":"ContainerStarted","Data":"b5b3b2984bc9c3292cb18b5bee2cd611539ae84d552b7a66bb680488b330acaf"} Nov 03 22:22:13.278291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:13.277602 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:13.278291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:13.277855 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:13.504750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:13.504669 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88khg" event={"ID":"eca1be8a-3b84-4ef6-9a53-28e189b9cae0","Type":"ContainerStarted","Data":"c3d4792009a6e423a3c8c5ee3c2e6168d45af27d838828d6cb8ba10dfa88a76a"} Nov 03 22:22:14.108571 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.108530 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:14.108971 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.108952 5051 scope.go:122] "RemoveContainer" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" Nov 03 22:22:14.109233 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:14.109209 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:14.512359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.512316 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8g4v" event={"ID":"f9d5d309-c79f-4f6d-bde8-d37d617eb61d","Type":"ContainerStarted","Data":"c8151fcacba48623d39d11e175f509b58cd278027313079a995649b7c69f97cc"} Nov 03 22:22:14.512806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.512390 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8g4v" event={"ID":"f9d5d309-c79f-4f6d-bde8-d37d617eb61d","Type":"ContainerStarted","Data":"3b2ec122a1be962d71a70969998a34a8ae3f98173b1d335e5c88b696063a75a2"} Nov 03 22:22:14.514393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.514361 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jdkv" event={"ID":"95598dfe-8d0c-4a37-9a8f-479bb69f07c0","Type":"ContainerStarted","Data":"e5487aa578acb4e31b2d8b75e50cf9aa1e294345c8cd25a2fa98cc994b7c721c"} Nov 03 22:22:14.516326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.516300 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" event={"ID":"a6e9c510-1535-4f63-adc0-f4de450b6900","Type":"ContainerStarted","Data":"3f8a759708f3af1f503938288e37c9d21231ba7112996c463261a4b89d321356"} Nov 03 22:22:14.516427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:14.516347 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8mrdt" event={"ID":"a6e9c510-1535-4f63-adc0-f4de450b6900","Type":"ContainerStarted","Data":"2fce7ce92d655d1fcb32b1a812be8c1add5f150ffd506fed9fbaffa84b062823"} Nov 03 22:22:15.249261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249220 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249276 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249325 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249367 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249405 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249417 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:15.249435 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249434 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57dd5bc9b9-hpc6k: secret "image-registry-tls" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.249459 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249485 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249502 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls podName:b6d0e4fb-a229-48a4-83ee-3f1f71d400da nodeName:}" failed. No retries permitted until 2026-11-03 22:22:23.249480357 +0000 UTC m=+16.986824588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls") pod "image-registry-57dd5bc9b9-hpc6k" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da") : secret "image-registry-tls" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249507 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84854fd764-cb9xd: secret "image-registry-tls" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249537 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249587 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls podName:b75e1082-c755-4119-9be9-328b7f4d91df nodeName:}" failed. No retries permitted until 2026-11-03 22:22:23.249544346 +0000 UTC m=+16.986888565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls") pod "image-registry-84854fd764-cb9xd" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df") : secret "image-registry-tls" not found Nov 03 22:22:15.249765 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.249608 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert podName:e0c6b7b5-cbbd-4486-a045-35348021f72e nodeName:}" failed. No retries permitted until 2026-11-03 22:22:23.249598256 +0000 UTC m=+16.986942465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-x6mwc" (UID: "e0c6b7b5-cbbd-4486-a045-35348021f72e") : secret "networking-console-plugin-cert" not found Nov 03 22:22:15.250296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.250269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f1540d-7f49-4949-9679-4c6ca2ba9de9-service-ca-bundle\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:15.252215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.252194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f1540d-7f49-4949-9679-4c6ca2ba9de9-metrics-certs\") pod \"router-default-79487cbf4c-klb6d\" (UID: \"17f1540d-7f49-4949-9679-4c6ca2ba9de9\") " pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:15.252318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.252219 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf65cc24-9b6c-4723-903e-c0b0a320ac8b-metrics-tls\") pod \"dns-default-xgvtf\" (UID: \"cf65cc24-9b6c-4723-903e-c0b0a320ac8b\") " pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:15.301463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.301436 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:15.314229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.314203 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:15.522133 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.522047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88khg" event={"ID":"eca1be8a-3b84-4ef6-9a53-28e189b9cae0","Type":"ContainerStarted","Data":"091bdbd0e6dc60e0ea82cb79fef2a391a8ce80783d01444f2f5216d1c01199ff"} Nov 03 22:22:15.789720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.789634 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79487cbf4c-klb6d"] Nov 03 22:22:15.793712 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:15.793681 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f1540d_7f49_4949_9679_4c6ca2ba9de9.slice/crio-7ca209ed4beaab70e12a7c503f180532981437c3417d0aaace1a4025fdbb272b WatchSource:0}: Error finding container 7ca209ed4beaab70e12a7c503f180532981437c3417d0aaace1a4025fdbb272b: Status 404 returned error can't find the container with id 7ca209ed4beaab70e12a7c503f180532981437c3417d0aaace1a4025fdbb272b Nov 03 22:22:15.793911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.793896 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgvtf"] Nov 03 22:22:15.836076 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:15.836048 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf65cc24_9b6c_4723_903e_c0b0a320ac8b.slice/crio-42bb644e97e4dbfe2912346cf87c0ebe636a60071479e528035096bba913b782 WatchSource:0}: Error finding container 42bb644e97e4dbfe2912346cf87c0ebe636a60071479e528035096bba913b782: Status 404 returned error can't find the container with id 42bb644e97e4dbfe2912346cf87c0ebe636a60071479e528035096bba913b782 Nov 03 22:22:15.844714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.844691 5051 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:15.845089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:15.845073 5051 scope.go:122] "RemoveContainer" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" Nov 03 22:22:15.845330 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:15.845309 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:16.527674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:16.527623 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgvtf" event={"ID":"cf65cc24-9b6c-4723-903e-c0b0a320ac8b","Type":"ContainerStarted","Data":"42bb644e97e4dbfe2912346cf87c0ebe636a60071479e528035096bba913b782"} Nov 03 22:22:16.531060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:16.531024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79487cbf4c-klb6d" event={"ID":"17f1540d-7f49-4949-9679-4c6ca2ba9de9","Type":"ContainerStarted","Data":"470851c69f50d1edae785664e5697b2707af9fd456f6f9ae22e9e904b8b77cc3"} Nov 03 22:22:16.531206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:16.531078 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79487cbf4c-klb6d" event={"ID":"17f1540d-7f49-4949-9679-4c6ca2ba9de9","Type":"ContainerStarted","Data":"7ca209ed4beaab70e12a7c503f180532981437c3417d0aaace1a4025fdbb272b"} Nov 03 22:22:17.302649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:17.302605 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:17.305800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:17.305775 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:18.405402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.405377 5051 scope.go:122] "RemoveContainer" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" Nov 03 22:22:18.539938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.539909 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgvtf" event={"ID":"cf65cc24-9b6c-4723-903e-c0b0a320ac8b","Type":"ContainerStarted","Data":"c52d2526aac1d5a132b9d2142e3e047ab8ec4515ca784e2298a7971f4054ed47"} Nov 03 22:22:18.540094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.539962 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgvtf" event={"ID":"cf65cc24-9b6c-4723-903e-c0b0a320ac8b","Type":"ContainerStarted","Data":"4e6e9ce7706dd3a91ed529df1dc0d09ab80daa29529fc7207421392c345df22f"} Nov 03 22:22:18.540094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.540026 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:18.544391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.544364 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:22:18.544786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.544771 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/2.log" Nov 03 22:22:18.544857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.544804 5051 generic.go:356] "Generic (PLEG): container finished" podID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerID="01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754" exitCode=255 Nov 03 22:22:18.544909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.544886 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:18.544966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.544880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerDied","Data":"01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754"} Nov 03 22:22:18.545088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.545019 5051 scope.go:122] "RemoveContainer" containerID="968bc5587749efe1c1c941f35f20b97f9b1c609384e43bc950d897a639c867cf" Nov 03 22:22:18.545250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:18.545230 5051 scope.go:122] "RemoveContainer" containerID="01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754" Nov 03 22:22:18.545452 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:18.545432 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:19.549605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:19.549574 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:22:21.621932 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:21.621894 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:22:22.312013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:22.311955 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:22.312196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:22.312111 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:22.313326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:22.313303 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79487cbf4c-klb6d" Nov 03 22:22:23.336311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.336270 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:23.336814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.336322 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:23.336814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.336377 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:23.339013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.338973 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0c6b7b5-cbbd-4486-a045-35348021f72e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-x6mwc\" (UID: \"e0c6b7b5-cbbd-4486-a045-35348021f72e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:23.339130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.338973 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"image-registry-84854fd764-cb9xd\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:23.339130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.339069 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"image-registry-57dd5bc9b9-hpc6k\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:23.402116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.402073 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:23.403838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.403815 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" Nov 03 22:22:23.413486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.413461 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:23.886206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.886171 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-x6mwc"] Nov 03 22:22:23.889677 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:23.889650 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6b7b5_cbbd_4486_a045_35348021f72e.slice/crio-033f1dd6f98051a969994b98ede142bf342dc9a7a2fcbd01b821d79dd64c7a60 WatchSource:0}: Error finding container 033f1dd6f98051a969994b98ede142bf342dc9a7a2fcbd01b821d79dd64c7a60: Status 404 returned error can't find the container with id 033f1dd6f98051a969994b98ede142bf342dc9a7a2fcbd01b821d79dd64c7a60 Nov 03 22:22:23.930252 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:23.930220 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e1082_c755_4119_9be9_328b7f4d91df.slice/crio-898f421251b64375417748c99ab7acda22341594a5c78ddbbedc103836bdb770 WatchSource:0}: Error finding container 898f421251b64375417748c99ab7acda22341594a5c78ddbbedc103836bdb770: Status 404 returned error can't find the container with id 898f421251b64375417748c99ab7acda22341594a5c78ddbbedc103836bdb770 Nov 03 22:22:23.935759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.935735 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:22:23.946129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:23.946095 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:22:23.949136 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:23.949108 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d0e4fb_a229_48a4_83ee_3f1f71d400da.slice/crio-3da00e2f895bdcf1d39e11c7991c244c4999aaaba1dde9163342d068660a1347 WatchSource:0}: Error finding container 3da00e2f895bdcf1d39e11c7991c244c4999aaaba1dde9163342d068660a1347: Status 404 returned error can't find the container with id 3da00e2f895bdcf1d39e11c7991c244c4999aaaba1dde9163342d068660a1347 Nov 03 22:22:24.109235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.109196 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:24.109650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.109629 5051 scope.go:122] "RemoveContainer" containerID="01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754" Nov 03 22:22:24.109853 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:24.109831 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:24.472266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.472232 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mlhk9" Nov 03 22:22:24.569419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.569382 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" event={"ID":"b75e1082-c755-4119-9be9-328b7f4d91df","Type":"ContainerStarted","Data":"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730"} Nov 03 22:22:24.569626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.569453 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" event={"ID":"b75e1082-c755-4119-9be9-328b7f4d91df","Type":"ContainerStarted","Data":"898f421251b64375417748c99ab7acda22341594a5c78ddbbedc103836bdb770"} Nov 03 22:22:24.569626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.569481 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:24.570919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.570899 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" event={"ID":"b6d0e4fb-a229-48a4-83ee-3f1f71d400da","Type":"ContainerStarted","Data":"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70"} Nov 03 22:22:24.571065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.570932 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" event={"ID":"b6d0e4fb-a229-48a4-83ee-3f1f71d400da","Type":"ContainerStarted","Data":"3da00e2f895bdcf1d39e11c7991c244c4999aaaba1dde9163342d068660a1347"} Nov 03 22:22:24.571065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.570975 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:24.572066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:24.572049 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" event={"ID":"e0c6b7b5-cbbd-4486-a045-35348021f72e","Type":"ContainerStarted","Data":"033f1dd6f98051a969994b98ede142bf342dc9a7a2fcbd01b821d79dd64c7a60"} Nov 03 22:22:25.577354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:25.577317 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-x6mwc" event={"ID":"e0c6b7b5-cbbd-4486-a045-35348021f72e","Type":"ContainerStarted","Data":"55456c733fa9abacc7684eec89c58356a0dbc2e239672aa75b9e2214160e83cb"} Nov 03 22:22:25.845525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:25.845491 5051 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:25.845884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:25.845871 5051 scope.go:122] "RemoveContainer" containerID="01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754" Nov 03 22:22:25.846112 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:25.846083 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-t8ztc_openshift-console-operator(0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" Nov 03 22:22:26.563909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.563872 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:22:26.590974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.590933 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64"] Nov 03 22:22:26.596445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.596412 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:26.644636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.644609 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c755c9-tlggr"] Nov 03 22:22:26.647807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.647789 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.666453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.666425 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/332253ec-6bf3-4703-9cb0-f87db7afa536-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zxg64\" (UID: \"332253ec-6bf3-4703-9cb0-f87db7afa536\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:26.690501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.690470 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 22:22:26.690661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.690558 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4cxhn\"" Nov 03 22:22:26.691550 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.691521 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64"] Nov 03 22:22:26.749291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.749260 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c755c9-tlggr"] Nov 03 22:22:26.767925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.767894 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe44950a-1b21-4b9e-8036-0e6c3a84532b-ca-trust-extracted\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768033 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-installation-pull-secrets\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768056 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-bound-sa-token\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768080 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-tls\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768152 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-image-registry-private-configuration\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768176 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-certificates\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768209 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-trusted-ca\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.768346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768325 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/332253ec-6bf3-4703-9cb0-f87db7afa536-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zxg64\" (UID: \"332253ec-6bf3-4703-9cb0-f87db7afa536\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:26.768388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.768352 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9vh\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-kube-api-access-sc9vh\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.770879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.770856 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/332253ec-6bf3-4703-9cb0-f87db7afa536-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zxg64\" (UID: \"332253ec-6bf3-4703-9cb0-f87db7afa536\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:26.869409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869374 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-installation-pull-secrets\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869409 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-bound-sa-token\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869435 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-tls\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869469 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-image-registry-private-configuration\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-certificates\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-trusted-ca\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869562 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9vh\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-kube-api-access-sc9vh\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.869657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.869602 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe44950a-1b21-4b9e-8036-0e6c3a84532b-ca-trust-extracted\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.870056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.870029 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe44950a-1b21-4b9e-8036-0e6c3a84532b-ca-trust-extracted\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.870506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.870488 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-certificates\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.871178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.871155 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe44950a-1b21-4b9e-8036-0e6c3a84532b-trusted-ca\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.872404 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.872383 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-registry-tls\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.872676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.872657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-image-registry-private-configuration\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.872728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.872713 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe44950a-1b21-4b9e-8036-0e6c3a84532b-installation-pull-secrets\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.909621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.909594 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:26.932560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.932525 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9vh\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-kube-api-access-sc9vh\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.932673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.932616 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe44950a-1b21-4b9e-8036-0e6c3a84532b-bound-sa-token\") pod \"image-registry-85c755c9-tlggr\" (UID: \"fe44950a-1b21-4b9e-8036-0e6c3a84532b\") " pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:26.958685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:26.958649 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:27.282270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.282195 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64"] Nov 03 22:22:27.328207 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:27.328178 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332253ec_6bf3_4703_9cb0_f87db7afa536.slice/crio-bd2080e09107e8111141f81d408e853ebd796a6d0b55badd92a806877c56d270 WatchSource:0}: Error finding container bd2080e09107e8111141f81d408e853ebd796a6d0b55badd92a806877c56d270: Status 404 returned error can't find the container with id bd2080e09107e8111141f81d408e853ebd796a6d0b55badd92a806877c56d270 Nov 03 22:22:27.341085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.341060 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c755c9-tlggr"] Nov 03 22:22:27.344502 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:27.344473 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe44950a_1b21_4b9e_8036_0e6c3a84532b.slice/crio-f11c6e33f1048d58d4b44e29aa287880fa135fa7e229859edf18896e9b0d67f5 WatchSource:0}: Error finding container f11c6e33f1048d58d4b44e29aa287880fa135fa7e229859edf18896e9b0d67f5: Status 404 returned error can't find the container with id f11c6e33f1048d58d4b44e29aa287880fa135fa7e229859edf18896e9b0d67f5 Nov 03 22:22:27.590467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.590372 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" event={"ID":"332253ec-6bf3-4703-9cb0-f87db7afa536","Type":"ContainerStarted","Data":"bd2080e09107e8111141f81d408e853ebd796a6d0b55badd92a806877c56d270"} Nov 03 22:22:27.592030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.591982 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c755c9-tlggr" event={"ID":"fe44950a-1b21-4b9e-8036-0e6c3a84532b","Type":"ContainerStarted","Data":"920d5fd6fad81c4aae53f328706a9c0e3765594ac6267eecc4c3d2ea2a964f68"} Nov 03 22:22:27.592487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.592051 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c755c9-tlggr" event={"ID":"fe44950a-1b21-4b9e-8036-0e6c3a84532b","Type":"ContainerStarted","Data":"f11c6e33f1048d58d4b44e29aa287880fa135fa7e229859edf18896e9b0d67f5"} Nov 03 22:22:27.592487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.592080 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:27.713317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.713253 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c755c9-tlggr" podStartSLOduration=1.713233562 podStartE2EDuration="1.713233562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:27.71238398 +0000 UTC m=+21.449728205" watchObservedRunningTime="2026-11-03 22:22:27.713233562 +0000 UTC m=+21.450577790" Nov 03 22:22:27.834504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:27.834477 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdcgs" Nov 03 22:22:28.551968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:28.551938 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xgvtf" Nov 03 22:22:29.602164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:29.602127 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" event={"ID":"332253ec-6bf3-4703-9cb0-f87db7afa536","Type":"ContainerStarted","Data":"2cbc91632b70af3b718cc882cd262f8d2d5b0d93c68a8ce39ad029b6cc9e34d0"} Nov 03 22:22:29.602557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:29.602350 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:29.607248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:29.607226 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" Nov 03 22:22:29.717084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:29.717036 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zxg64" podStartSLOduration=2.320484108 podStartE2EDuration="3.717021402s" totalImagesPullingTime="1.396537294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:29.715884246 +0000 UTC m=+23.453228471" watchObservedRunningTime="2026-11-03 22:22:29.717021402 +0000 UTC m=+23.454365621" Nov 03 22:22:37.117792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.117756 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqdzw"] Nov 03 22:22:37.159903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.159877 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.264764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.264730 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:22:37.265026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.264854 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:22:37.265026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.264924 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:22:37.265139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.265039 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:22:37.265139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.265051 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q5zlz\"" Nov 03 22:22:37.272304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-root\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272315 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-tls\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272344 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-metrics-client-ca\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272450 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-sys\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272476 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272513 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-textfile\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272634 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-wtmp\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.272746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.272701 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vqc\" (UniqueName: \"kubernetes.io/projected/0895fe48-a6af-47c0-894c-8d0940027e95-kube-api-access-g2vqc\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374119 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-textfile\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374159 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-wtmp\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374193 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vqc\" (UniqueName: \"kubernetes.io/projected/0895fe48-a6af-47c0-894c-8d0940027e95-kube-api-access-g2vqc\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374224 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-root\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374240 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-tls\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374254 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-metrics-client-ca\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374271 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-sys\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374306 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374341 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-root\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-wtmp\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374384 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0895fe48-a6af-47c0-894c-8d0940027e95-sys\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374564 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-textfile\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.374942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.374920 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.375092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.375073 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0895fe48-a6af-47c0-894c-8d0940027e95-metrics-client-ca\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.376767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.376746 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-tls\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.376875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.376797 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0895fe48-a6af-47c0-894c-8d0940027e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.434906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.434878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vqc\" (UniqueName: \"kubernetes.io/projected/0895fe48-a6af-47c0-894c-8d0940027e95-kube-api-access-g2vqc\") pod \"node-exporter-tqdzw\" (UID: \"0895fe48-a6af-47c0-894c-8d0940027e95\") " pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.508581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.508550 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqdzw" Nov 03 22:22:37.533459 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:22:37.533422 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0895fe48_a6af_47c0_894c_8d0940027e95.slice/crio-940e2b031e120b002ba1fb748e49461f833287ea0133f082c7a43acd237b0436 WatchSource:0}: Error finding container 940e2b031e120b002ba1fb748e49461f833287ea0133f082c7a43acd237b0436: Status 404 returned error can't find the container with id 940e2b031e120b002ba1fb748e49461f833287ea0133f082c7a43acd237b0436 Nov 03 22:22:37.635305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:37.635233 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqdzw" event={"ID":"0895fe48-a6af-47c0-894c-8d0940027e95","Type":"ContainerStarted","Data":"940e2b031e120b002ba1fb748e49461f833287ea0133f082c7a43acd237b0436"} Nov 03 22:22:38.471536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:38.471507 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wmt2m" Nov 03 22:22:40.400077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.400046 5051 scope.go:122] "RemoveContainer" containerID="01ea0d7134ea89623a7f3cd9e8852582e02a96108a30e299ef231d0c66261754" Nov 03 22:22:40.649967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.649934 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:22:40.650193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.650071 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" event={"ID":"0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3","Type":"ContainerStarted","Data":"19e580168a31bf298126bfddee3ca701c1dff76a153056c9c62978c7ea91d4cc"} Nov 03 22:22:40.650402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.650372 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:40.651842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.651817 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-t8ztc container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Nov 03 22:22:40.651943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.651857 5051 generic.go:356] "Generic (PLEG): container finished" podID="0895fe48-a6af-47c0-894c-8d0940027e95" containerID="eb9651e00239d3ead91ba2ef7bfa2ea85b1b0d0b1a60c7c69c492ec41bbf0035" exitCode=0 Nov 03 22:22:40.651943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.651862 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" podUID="0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Nov 03 22:22:40.651943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:40.651883 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqdzw" event={"ID":"0895fe48-a6af-47c0-894c-8d0940027e95","Type":"ContainerDied","Data":"eb9651e00239d3ead91ba2ef7bfa2ea85b1b0d0b1a60c7c69c492ec41bbf0035"} Nov 03 22:22:41.658625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:41.658587 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqdzw" event={"ID":"0895fe48-a6af-47c0-894c-8d0940027e95","Type":"ContainerStarted","Data":"ad1513a8d2fbb9319342ae9ce56a3f5f158f06dc3fabd664505193488819b160"} Nov 03 22:22:41.659153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:41.658649 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqdzw" event={"ID":"0895fe48-a6af-47c0-894c-8d0940027e95","Type":"ContainerStarted","Data":"7828b2635e656b96d62364e2f62d04042163d8835c607aeb874364a2350676b1"} Nov 03 22:22:41.662689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:41.662663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-t8ztc" Nov 03 22:22:41.780066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:41.779981 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqdzw" podStartSLOduration=2.6287228110000003 podStartE2EDuration="4.779960936s" totalImagesPullingTime="2.059296685s" totalInitContainerRuntime="91.94144ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.779783088 +0000 UTC m=+35.517127314" watchObservedRunningTime="2026-11-03 22:22:41.779960936 +0000 UTC m=+35.517305161" Nov 03 22:22:45.581809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:45.581732 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:22:46.590380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:46.590353 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:48.601689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:48.601660 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c755c9-tlggr" Nov 03 22:22:48.772526 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:48.772491 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:22:50.754714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:50.754678 5051 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mlhk9_661b34e8-d977-46ae-a3c6-6f09a1182f8d/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:51.610128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.610047 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" podUID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" containerName="registry" containerID="cri-o://13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70" gracePeriod=30 Nov 03 22:22:51.625167 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:51.625137 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:22:51.869582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.869520 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:51.900275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900250 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900304 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900335 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900393 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900459 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900492 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900524 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900562 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") pod \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\" (UID: \"b6d0e4fb-a229-48a4-83ee-3f1f71d400da\") " Nov 03 22:22:51.900897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900745 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:51.901041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.900982 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:22:51.903006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.902966 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:51.903134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.903038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "kube-api-access-zjvnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:51.903417 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.903392 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:22:51.903417 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.903405 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:51.903561 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.903543 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:22:51.908318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:51.908291 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted" pod "b6d0e4fb-a229-48a4-83ee-3f1f71d400da" (UID: "b6d0e4fb-a229-48a4-83ee-3f1f71d400da"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:22:52.001701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001670 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-trusted-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001699 5051 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-ca-trust-extracted\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001708 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvnn\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-kube-api-access-zjvnn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001719 5051 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-installation-pull-secrets\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001729 5051 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-image-registry-private-configuration\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001738 5051 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-bound-sa-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001746 5051 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-certificates\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.001926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.001755 5051 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6d0e4fb-a229-48a4-83ee-3f1f71d400da-registry-tls\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:22:52.706928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.706846 5051 generic.go:356] "Generic (PLEG): container finished" podID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" containerID="13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70" exitCode=0 Nov 03 22:22:52.706928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.706904 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" Nov 03 22:22:52.707147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.706922 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" event={"ID":"b6d0e4fb-a229-48a4-83ee-3f1f71d400da","Type":"ContainerDied","Data":"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70"} Nov 03 22:22:52.707147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.707015 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dd5bc9b9-hpc6k" event={"ID":"b6d0e4fb-a229-48a4-83ee-3f1f71d400da","Type":"ContainerDied","Data":"3da00e2f895bdcf1d39e11c7991c244c4999aaaba1dde9163342d068660a1347"} Nov 03 22:22:52.707147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.707044 5051 scope.go:122] "RemoveContainer" containerID="13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70" Nov 03 22:22:52.715654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.715639 5051 scope.go:122] "RemoveContainer" containerID="13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70" Nov 03 22:22:52.715937 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:22:52.715919 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70\": container with ID starting with 13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70 not found: ID does not exist" containerID="13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70" Nov 03 22:22:52.715979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.715946 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70"} err="failed to get container status \"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70\": rpc error: code = NotFound desc = could not find container \"13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70\": container with ID starting with 13f439bf667146849507af7247f479f1a995051dea415b1bd71dd9612d422a70 not found: ID does not exist" Nov 03 22:22:52.923628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.923598 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:22:52.973249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:52.973182 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57dd5bc9b9-hpc6k"] Nov 03 22:22:54.405476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:22:54.405446 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" path="/var/lib/kubelet/pods/b6d0e4fb-a229-48a4-83ee-3f1f71d400da/volumes" Nov 03 22:23:13.792928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:13.792891 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" podUID="b75e1082-c755-4119-9be9-328b7f4d91df" containerName="registry" containerID="cri-o://0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730" gracePeriod=30 Nov 03 22:23:14.034340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.034315 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:23:14.089903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.089833 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.089903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.089885 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.089905 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.089936 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.089968 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.090015 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.090048 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.090229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.090084 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") pod \"b75e1082-c755-4119-9be9-328b7f4d91df\" (UID: \"b75e1082-c755-4119-9be9-328b7f4d91df\") " Nov 03 22:23:14.091223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.091187 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:14.092467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.090607 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:14.095040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.093144 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:14.098310 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.093389 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:14.098687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.098644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "kube-api-access-jr9fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:14.098830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.098800 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:14.099028 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.098972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:14.100204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.100096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted" pod "b75e1082-c755-4119-9be9-328b7f4d91df" (UID: "b75e1082-c755-4119-9be9-328b7f4d91df"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:14.190953 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.190914 5051 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-registry-certificates\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.190953 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.190951 5051 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-bound-sa-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.190965 5051 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-registry-tls\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.190978 5051 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-image-registry-private-configuration\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.191024 5051 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b75e1082-c755-4119-9be9-328b7f4d91df-installation-pull-secrets\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.191036 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9fh\" (UniqueName: \"kubernetes.io/projected/b75e1082-c755-4119-9be9-328b7f4d91df-kube-api-access-jr9fh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.191048 5051 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b75e1082-c755-4119-9be9-328b7f4d91df-ca-trust-extracted\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.191371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.191059 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b75e1082-c755-4119-9be9-328b7f4d91df-trusted-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:14.793159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.793063 5051 generic.go:356] "Generic (PLEG): container finished" podID="b75e1082-c755-4119-9be9-328b7f4d91df" containerID="0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730" exitCode=0 Nov 03 22:23:14.793159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.793132 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" Nov 03 22:23:14.793159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.793137 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" event={"ID":"b75e1082-c755-4119-9be9-328b7f4d91df","Type":"ContainerDied","Data":"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730"} Nov 03 22:23:14.793724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.793185 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84854fd764-cb9xd" event={"ID":"b75e1082-c755-4119-9be9-328b7f4d91df","Type":"ContainerDied","Data":"898f421251b64375417748c99ab7acda22341594a5c78ddbbedc103836bdb770"} Nov 03 22:23:14.793724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.793206 5051 scope.go:122] "RemoveContainer" containerID="0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730" Nov 03 22:23:14.801290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.801266 5051 scope.go:122] "RemoveContainer" containerID="0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730" Nov 03 22:23:14.801571 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:23:14.801551 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730\": container with ID starting with 0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730 not found: ID does not exist" containerID="0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730" Nov 03 22:23:14.801639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:14.801587 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730"} err="failed to get container status \"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730\": rpc error: code = NotFound desc = could not find container \"0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730\": container with ID starting with 0a078c7a8bd1079aa14357cb0f3232d549d1cd6df58a07a0dbbebdacda3de730 not found: ID does not exist" Nov 03 22:23:15.013245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:15.013214 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:23:15.063190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:15.063113 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84854fd764-cb9xd"] Nov 03 22:23:16.406132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:16.406101 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75e1082-c755-4119-9be9-328b7f4d91df" path="/var/lib/kubelet/pods/b75e1082-c755-4119-9be9-328b7f4d91df/volumes" Nov 03 22:23:20.822256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:20.822222 5051 generic.go:356] "Generic (PLEG): container finished" podID="e906de24-2e38-4a47-81cd-581b7c617454" containerID="1afa1b962c64694572f82b3f3d72cc113e056dbc8da0af3978b19cf7b5d1f499" exitCode=0 Nov 03 22:23:20.822623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:20.822299 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plm6l" event={"ID":"e906de24-2e38-4a47-81cd-581b7c617454","Type":"ContainerDied","Data":"1afa1b962c64694572f82b3f3d72cc113e056dbc8da0af3978b19cf7b5d1f499"} Nov 03 22:23:20.822659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:20.822641 5051 scope.go:122] "RemoveContainer" containerID="1afa1b962c64694572f82b3f3d72cc113e056dbc8da0af3978b19cf7b5d1f499" Nov 03 22:23:21.622868 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:23:21.622832 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:23:21.827905 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:21.827869 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-plm6l" event={"ID":"e906de24-2e38-4a47-81cd-581b7c617454","Type":"ContainerStarted","Data":"ff75a0388c3f58eddd9b28cff9dddc8fd292dd1080052876f30593c886300b43"} Nov 03 22:23:21.995318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:21.995215 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:23:21.995506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:21.995479 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" podUID="e57fcd21-a810-4e2f-880b-d99b80541c6a" containerName="insights-gathering" containerID="cri-o://cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a" gracePeriod=30 Nov 03 22:23:22.262212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.262187 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:23:22.358477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358441 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") pod \"e57fcd21-a810-4e2f-880b-d99b80541c6a\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " Nov 03 22:23:22.358677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358559 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") pod \"e57fcd21-a810-4e2f-880b-d99b80541c6a\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " Nov 03 22:23:22.358677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358591 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") pod \"e57fcd21-a810-4e2f-880b-d99b80541c6a\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " Nov 03 22:23:22.358677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358620 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") pod \"e57fcd21-a810-4e2f-880b-d99b80541c6a\" (UID: \"e57fcd21-a810-4e2f-880b-d99b80541c6a\") " Nov 03 22:23:22.358860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358830 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle" pod "e57fcd21-a810-4e2f-880b-d99b80541c6a" (UID: "e57fcd21-a810-4e2f-880b-d99b80541c6a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:22.358904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358888 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path" pod "e57fcd21-a810-4e2f-880b-d99b80541c6a" (UID: "e57fcd21-a810-4e2f-880b-d99b80541c6a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:22.358957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.358939 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle" pod "e57fcd21-a810-4e2f-880b-d99b80541c6a" (UID: "e57fcd21-a810-4e2f-880b-d99b80541c6a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:22.361028 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.361007 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg" pod "e57fcd21-a810-4e2f-880b-d99b80541c6a" (UID: "e57fcd21-a810-4e2f-880b-d99b80541c6a"). InnerVolumeSpecName "kube-api-access-b5rhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:22.459613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.459574 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-trusted-ca-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:22.459613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.459604 5051 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e57fcd21-a810-4e2f-880b-d99b80541c6a-archives-path\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:22.459613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.459614 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5rhg\" (UniqueName: \"kubernetes.io/projected/e57fcd21-a810-4e2f-880b-d99b80541c6a-kube-api-access-b5rhg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:22.459836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.459625 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57fcd21-a810-4e2f-880b-d99b80541c6a-service-ca-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:22.832481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.832447 5051 generic.go:356] "Generic (PLEG): container finished" podID="e57fcd21-a810-4e2f-880b-d99b80541c6a" containerID="cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a" exitCode=2 Nov 03 22:23:22.833080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.832508 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" Nov 03 22:23:22.833080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.832546 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" event={"ID":"e57fcd21-a810-4e2f-880b-d99b80541c6a","Type":"ContainerDied","Data":"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a"} Nov 03 22:23:22.833080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.832596 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lkvmc-bgpw2" event={"ID":"e57fcd21-a810-4e2f-880b-d99b80541c6a","Type":"ContainerDied","Data":"9b4e6a86c08e1d39eb96169894e5e1a3fd42581a4b6741c8ee47e24ee2d436ce"} Nov 03 22:23:22.833080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.832616 5051 scope.go:122] "RemoveContainer" containerID="cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a" Nov 03 22:23:22.840596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.840579 5051 scope.go:122] "RemoveContainer" containerID="cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a" Nov 03 22:23:22.840873 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:23:22.840856 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a\": container with ID starting with cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a not found: ID does not exist" containerID="cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a" Nov 03 22:23:22.840914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.840882 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a"} err="failed to get container status \"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a\": rpc error: code = NotFound desc = could not find container \"cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a\": container with ID starting with cff615c99d77e4c27cda7612333048d624e37f15fc4d50d616edfd633e58130a not found: ID does not exist" Nov 03 22:23:22.994650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.994618 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:23:22.998206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:22.998183 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-lkvmc-bgpw2"] Nov 03 22:23:24.405301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:24.405266 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57fcd21-a810-4e2f-880b-d99b80541c6a" path="/var/lib/kubelet/pods/e57fcd21-a810-4e2f-880b-d99b80541c6a/volumes" Nov 03 22:23:26.850814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:26.850731 5051 generic.go:356] "Generic (PLEG): container finished" podID="b5d6573e-1446-4ea2-b6f1-9cf9e19a161d" containerID="7d9cf633c8f8b07bbcdd84b62cf5c948f0881cf2b9a00aba610361ba0b9da0bc" exitCode=0 Nov 03 22:23:26.850814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:26.850806 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" event={"ID":"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d","Type":"ContainerDied","Data":"7d9cf633c8f8b07bbcdd84b62cf5c948f0881cf2b9a00aba610361ba0b9da0bc"} Nov 03 22:23:26.851251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:26.851132 5051 scope.go:122] "RemoveContainer" containerID="7d9cf633c8f8b07bbcdd84b62cf5c948f0881cf2b9a00aba610361ba0b9da0bc" Nov 03 22:23:27.858061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:27.858023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6btx" event={"ID":"b5d6573e-1446-4ea2-b6f1-9cf9e19a161d","Type":"ContainerStarted","Data":"9247bcec1c3dbc2d00add4600f26ae218bef51f97ef817195a2f8625e8ff85f6"} Nov 03 22:23:28.863557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:28.863524 5051 generic.go:356] "Generic (PLEG): container finished" podID="6f77da67-81d4-4a4e-a92b-c69542057b14" containerID="fce479c91d88ce27080ab8358a108e2ea59c2d46feda21fd8833d405fdb4a100" exitCode=0 Nov 03 22:23:28.863957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:28.863598 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" event={"ID":"6f77da67-81d4-4a4e-a92b-c69542057b14","Type":"ContainerDied","Data":"fce479c91d88ce27080ab8358a108e2ea59c2d46feda21fd8833d405fdb4a100"} Nov 03 22:23:28.863957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:28.863919 5051 scope.go:122] "RemoveContainer" containerID="fce479c91d88ce27080ab8358a108e2ea59c2d46feda21fd8833d405fdb4a100" Nov 03 22:23:29.870714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:23:29.870673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ckj7j" event={"ID":"6f77da67-81d4-4a4e-a92b-c69542057b14","Type":"ContainerStarted","Data":"9114d0fd361f8fae6ecf3cdce299569690bcb3f501d28cfe958ea0210c345913"} Nov 03 22:23:51.623323 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:23:51.623284 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:24:21.622267 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:24:21.622180 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:24:51.621964 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:24:51.621925 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:25:21.621655 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:25:21.621604 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:25:51.623359 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:25:51.623282 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:26:21.622668 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:26:21.622629 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:26:51.622286 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:26:51.622251 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:27:00.795532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795497 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv"] Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795874 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" containerName="registry" Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795890 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" containerName="registry" Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795921 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e57fcd21-a810-4e2f-880b-d99b80541c6a" containerName="insights-gathering" Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795929 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57fcd21-a810-4e2f-880b-d99b80541c6a" containerName="insights-gathering" Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795943 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b75e1082-c755-4119-9be9-328b7f4d91df" containerName="registry" Nov 03 22:27:00.796024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.795950 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75e1082-c755-4119-9be9-328b7f4d91df" containerName="registry" Nov 03 22:27:00.796238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.796069 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d0e4fb-a229-48a4-83ee-3f1f71d400da" containerName="registry" Nov 03 22:27:00.796238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.796083 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57fcd21-a810-4e2f-880b-d99b80541c6a" containerName="insights-gathering" Nov 03 22:27:00.796238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.796090 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75e1082-c755-4119-9be9-328b7f4d91df" containerName="registry" Nov 03 22:27:00.799159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.799141 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.880315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.880285 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:27:00.880315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.880285 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-wv88x\"" Nov 03 22:27:00.905348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.905316 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv"] Nov 03 22:27:00.907370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907343 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907382 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907413 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907482 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf28985-5cc8-42ba-b374-f44d7704a0d3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907529 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdfj\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-kube-api-access-xcdfj\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907604 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907644 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907669 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:00.907823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:00.907734 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.008751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008713 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.008751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008751 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf28985-5cc8-42ba-b374-f44d7704a0d3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008770 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdfj\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-kube-api-access-xcdfj\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008792 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008857 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008913 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.008952 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.009301 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.009384 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.009445 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.009496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.009591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.009575 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf28985-5cc8-42ba-b374-f44d7704a0d3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.011431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.011411 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.011556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.011536 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.069923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.069843 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdfj\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-kube-api-access-xcdfj\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.069923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.069839 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf28985-5cc8-42ba-b374-f44d7704a0d3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv\" (UID: \"abf28985-5cc8-42ba-b374-f44d7704a0d3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.112363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.112328 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:01.539923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.539890 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv"] Nov 03 22:27:01.542888 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:01.542862 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf28985_5cc8_42ba_b374_f44d7704a0d3.slice/crio-0dff37126d7ab7b02e53b11553b4bec8a85b2296b0c7379ada479ba5ee4ab810 WatchSource:0}: Error finding container 0dff37126d7ab7b02e53b11553b4bec8a85b2296b0c7379ada479ba5ee4ab810: Status 404 returned error can't find the container with id 0dff37126d7ab7b02e53b11553b4bec8a85b2296b0c7379ada479ba5ee4ab810 Nov 03 22:27:01.706656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:01.706624 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" event={"ID":"abf28985-5cc8-42ba-b374-f44d7704a0d3","Type":"ContainerStarted","Data":"0dff37126d7ab7b02e53b11553b4bec8a85b2296b0c7379ada479ba5ee4ab810"} Nov 03 22:27:03.742340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.742310 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9"] Nov 03 22:27:03.745915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.745892 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.779767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.779733 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf"] Nov 03 22:27:03.783396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.783369 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:03.827880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.827849 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:27:03.827880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.827894 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:27:03.829520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.829478 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9"] Nov 03 22:27:03.835511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.835474 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.835697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.835538 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.835697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.835645 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqn7k\" (UniqueName: \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.852023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.851231 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9k8h2\"" Nov 03 22:27:03.854227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.853170 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz"] Nov 03 22:27:03.858192 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.858167 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf"] Nov 03 22:27:03.858325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.858210 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f"] Nov 03 22:27:03.858389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.858322 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:03.866002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.865967 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:03.890211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.890177 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz"] Nov 03 22:27:03.908555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.908527 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-4rmt2\"" Nov 03 22:27:03.910394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.910360 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jj5pb\"" Nov 03 22:27:03.917977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.917943 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qst65\"" Nov 03 22:27:03.936502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936468 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:03.936686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936549 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.936686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.936686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936619 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqn7k\" (UniqueName: \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.936686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936662 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mwc\" (UniqueName: \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:03.937036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.936982 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:03.937036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.937027 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.937191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.937076 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:03.955718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:03.955679 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f"] Nov 03 22:27:04.000394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.000323 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqn7k\" (UniqueName: \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:04.038593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.038553 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktz6j\" (UniqueName: \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.038774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.038621 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.038774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.038741 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.038871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.038790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.038871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.038855 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mwc\" (UniqueName: \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.039107 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039087 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.039208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039183 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.039247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039226 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.039305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.039359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039323 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7wd\" (UniqueName: \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.039512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.039486 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.065806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.065777 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:04.100566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.100539 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mwc\" (UniqueName: \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.100723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.100666 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:04.139890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.139855 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.140096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.139902 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7wd\" (UniqueName: \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.140096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.139957 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktz6j\" (UniqueName: \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.140096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.140070 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.140234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.140105 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.140234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.140150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.140639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.140604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.141770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.141696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.141770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.141694 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.142055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.142031 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.203849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.203821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktz6j\" (UniqueName: \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.204421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.204401 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7wd\" (UniqueName: \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.488191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.488158 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:04.488372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.488354 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9"] Nov 03 22:27:04.491506 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:04.491467 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbe982a_7cfb_4c2f_935d_5a625a6e5044.slice/crio-0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb WatchSource:0}: Error finding container 0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb: Status 404 returned error can't find the container with id 0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb Nov 03 22:27:04.495657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.495641 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:04.502565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.502539 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf"] Nov 03 22:27:04.506653 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:04.506574 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d7cf55_6057_4806_b438_8adcbd7e767b.slice/crio-650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826 WatchSource:0}: Error finding container 650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826: Status 404 returned error can't find the container with id 650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826 Nov 03 22:27:04.723822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.723782 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" event={"ID":"38d7cf55-6057-4806-b438-8adcbd7e767b","Type":"ContainerStarted","Data":"650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826"} Nov 03 22:27:04.725250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.725225 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerStarted","Data":"0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb"} Nov 03 22:27:04.877008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.876954 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz"] Nov 03 22:27:04.880425 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:04.880393 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c2cd72_41b8_4a4e_a217_4d47a35e5f3c.slice/crio-89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c WatchSource:0}: Error finding container 89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c: Status 404 returned error can't find the container with id 89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c Nov 03 22:27:04.925846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:04.925815 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f"] Nov 03 22:27:04.929534 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:04.929507 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638b03b2_d294_45c2_9bb2_a6e169eb0082.slice/crio-d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703 WatchSource:0}: Error finding container d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703: Status 404 returned error can't find the container with id d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703 Nov 03 22:27:05.347443 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.347409 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:05.347549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.347483 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:05.347549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.347527 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:05.733941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.733904 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" event={"ID":"abf28985-5cc8-42ba-b374-f44d7704a0d3","Type":"ContainerStarted","Data":"6b04659460f5de7287c299943d0cf09f021934a267abac9c14bfdee96842170b"} Nov 03 22:27:05.738699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.738663 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" event={"ID":"638b03b2-d294-45c2-9bb2-a6e169eb0082","Type":"ContainerStarted","Data":"d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703"} Nov 03 22:27:05.740325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.740279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" event={"ID":"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c","Type":"ContainerStarted","Data":"89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c"} Nov 03 22:27:05.859908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:05.859845 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" podStartSLOduration=2.05809158 podStartE2EDuration="5.859826657s" totalImagesPullingTime="3.801735077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:05.859430562 +0000 UTC m=+299.596774788" watchObservedRunningTime="2026-11-03 22:27:05.859826657 +0000 UTC m=+299.597170899" Nov 03 22:27:06.113268 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.113229 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:06.119450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.119394 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:06.448483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.448391 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:27:06.448770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.448749 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:27:06.747258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.747182 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:06.748346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:06.748320 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv" Nov 03 22:27:14.795077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.795041 5051 generic.go:356] "Generic (PLEG): container finished" podID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerID="78ba86b8e5dc6c2986564df6ad83cd3430f0bf34de15ad98183311d3241f3106" exitCode=0 Nov 03 22:27:14.795653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.795126 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" event={"ID":"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c","Type":"ContainerDied","Data":"78ba86b8e5dc6c2986564df6ad83cd3430f0bf34de15ad98183311d3241f3106"} Nov 03 22:27:14.796379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.796359 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:27:14.796823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.796799 5051 generic.go:356] "Generic (PLEG): container finished" podID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerID="004dd8b4622c58dc53e57980baa5ac7583daf04ecb4cee66248e3becbd22f83f" exitCode=0 Nov 03 22:27:14.796914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.796877 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerDied","Data":"004dd8b4622c58dc53e57980baa5ac7583daf04ecb4cee66248e3becbd22f83f"} Nov 03 22:27:14.798488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.798468 5051 generic.go:356] "Generic (PLEG): container finished" podID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerID="596211dade656d30978a28e527c9f1a9429bcddbf95eea06f14be2d7c45ae1bf" exitCode=0 Nov 03 22:27:14.798586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.798539 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" event={"ID":"38d7cf55-6057-4806-b438-8adcbd7e767b","Type":"ContainerDied","Data":"596211dade656d30978a28e527c9f1a9429bcddbf95eea06f14be2d7c45ae1bf"} Nov 03 22:27:14.800036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.800014 5051 generic.go:356] "Generic (PLEG): container finished" podID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerID="b600b88829b74403adc84259ceba6126a82cab6124357fe370f053a518108c9b" exitCode=0 Nov 03 22:27:14.800110 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:14.800084 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" event={"ID":"638b03b2-d294-45c2-9bb2-a6e169eb0082","Type":"ContainerDied","Data":"b600b88829b74403adc84259ceba6126a82cab6124357fe370f053a518108c9b"} Nov 03 22:27:16.809799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.809762 5051 generic.go:356] "Generic (PLEG): container finished" podID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerID="55ce92e4caa3c2ad42f7d25ea6dd65290e7d503049b75acb5adeaee3570ff8cc" exitCode=0 Nov 03 22:27:16.810349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.809846 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" event={"ID":"38d7cf55-6057-4806-b438-8adcbd7e767b","Type":"ContainerDied","Data":"55ce92e4caa3c2ad42f7d25ea6dd65290e7d503049b75acb5adeaee3570ff8cc"} Nov 03 22:27:16.811860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.811840 5051 generic.go:356] "Generic (PLEG): container finished" podID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerID="fdc6c5813abcb13ce0f2a51026850f3f3ecfe3d1d612d47459d4ea3593cd6b80" exitCode=0 Nov 03 22:27:16.811944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.811925 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" event={"ID":"638b03b2-d294-45c2-9bb2-a6e169eb0082","Type":"ContainerDied","Data":"fdc6c5813abcb13ce0f2a51026850f3f3ecfe3d1d612d47459d4ea3593cd6b80"} Nov 03 22:27:16.814410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.814389 5051 generic.go:356] "Generic (PLEG): container finished" podID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerID="b24773484b5c91dd1cbaf8e30353c40812348b32b90e8c0e65e98cd95a617172" exitCode=0 Nov 03 22:27:16.814498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.814471 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" event={"ID":"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c","Type":"ContainerDied","Data":"b24773484b5c91dd1cbaf8e30353c40812348b32b90e8c0e65e98cd95a617172"} Nov 03 22:27:16.816314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.816294 5051 generic.go:356] "Generic (PLEG): container finished" podID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerID="387798d4e8076b9d08f33567ec42ca4d2f1c128c41110336c67127ba7d41cdc2" exitCode=0 Nov 03 22:27:16.816390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:16.816337 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerDied","Data":"387798d4e8076b9d08f33567ec42ca4d2f1c128c41110336c67127ba7d41cdc2"} Nov 03 22:27:21.635811 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:27:21.635770 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:27:26.889409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:26.889370 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerStarted","Data":"bf320b9707b492bf4a48c33e324e33dc9aa99a377f084deca9c37fef467e8a36"} Nov 03 22:27:27.035327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.035233 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" podStartSLOduration=3.480305628 podStartE2EDuration="24.035217688s" totalImagesPullingTime="20.55491206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:27.033398139 +0000 UTC m=+320.770742365" watchObservedRunningTime="2026-11-03 22:27:27.035217688 +0000 UTC m=+320.772561923" Nov 03 22:27:27.895661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.895626 5051 generic.go:356] "Generic (PLEG): container finished" podID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerID="bf320b9707b492bf4a48c33e324e33dc9aa99a377f084deca9c37fef467e8a36" exitCode=0 Nov 03 22:27:27.896111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.895730 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerDied","Data":"bf320b9707b492bf4a48c33e324e33dc9aa99a377f084deca9c37fef467e8a36"} Nov 03 22:27:27.897817 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.897792 5051 generic.go:356] "Generic (PLEG): container finished" podID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerID="7af408b59940bf9f6c7bd4147ebd4344da5fca474166a6dae1344c3403453ee8" exitCode=0 Nov 03 22:27:27.897942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.897860 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" event={"ID":"38d7cf55-6057-4806-b438-8adcbd7e767b","Type":"ContainerDied","Data":"7af408b59940bf9f6c7bd4147ebd4344da5fca474166a6dae1344c3403453ee8"} Nov 03 22:27:27.899983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.899961 5051 generic.go:356] "Generic (PLEG): container finished" podID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerID="6614aba5cb95af350dd46585ab770355aeca08e9d2525256dc882c4221d49b13" exitCode=0 Nov 03 22:27:27.900104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.900031 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" event={"ID":"638b03b2-d294-45c2-9bb2-a6e169eb0082","Type":"ContainerDied","Data":"6614aba5cb95af350dd46585ab770355aeca08e9d2525256dc882c4221d49b13"} Nov 03 22:27:27.902118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.902098 5051 generic.go:356] "Generic (PLEG): container finished" podID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerID="ded2a65736246ed78365b91ddffddde8026b12d1838a860b4f1763c4806f0efe" exitCode=0 Nov 03 22:27:27.902247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:27.902135 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" event={"ID":"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c","Type":"ContainerDied","Data":"ded2a65736246ed78365b91ddffddde8026b12d1838a860b4f1763c4806f0efe"} Nov 03 22:27:29.064789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.064761 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:29.102485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.102463 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:29.105605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.105588 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:29.108860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.108842 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:29.180778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180690 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\" (UniqueName: \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\") pod \"638b03b2-d294-45c2-9bb2-a6e169eb0082\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " Nov 03 22:27:29.180778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180767 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\") pod \"638b03b2-d294-45c2-9bb2-a6e169eb0082\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180800 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\") pod \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180823 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\") pod \"38d7cf55-6057-4806-b438-8adcbd7e767b\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180858 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\" (UniqueName: \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\") pod \"38d7cf55-6057-4806-b438-8adcbd7e767b\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180889 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\") pod \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180916 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\" (UniqueName: \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\") pod \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.180961 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\" (UniqueName: \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\") pod \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " Nov 03 22:27:29.181031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.181020 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\") pod \"38d7cf55-6057-4806-b438-8adcbd7e767b\" (UID: \"38d7cf55-6057-4806-b438-8adcbd7e767b\") " Nov 03 22:27:29.181362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.181053 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\") pod \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\" (UID: \"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c\") " Nov 03 22:27:29.181362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.181097 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\") pod \"638b03b2-d294-45c2-9bb2-a6e169eb0082\" (UID: \"638b03b2-d294-45c2-9bb2-a6e169eb0082\") " Nov 03 22:27:29.181362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.181127 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\") pod \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\" (UID: \"6dbe982a-7cfb-4c2f-935d-5a625a6e5044\") " Nov 03 22:27:29.183473 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.183400 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle" pod "6dbe982a-7cfb-4c2f-935d-5a625a6e5044" (UID: "6dbe982a-7cfb-4c2f-935d-5a625a6e5044"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.184040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.183911 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle" pod "38d7cf55-6057-4806-b438-8adcbd7e767b" (UID: "38d7cf55-6057-4806-b438-8adcbd7e767b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.184371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.184338 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j" pod "638b03b2-d294-45c2-9bb2-a6e169eb0082" (UID: "638b03b2-d294-45c2-9bb2-a6e169eb0082"). InnerVolumeSpecName "kube-api-access-ktz6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:29.184480 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.184410 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc" pod "38d7cf55-6057-4806-b438-8adcbd7e767b" (UID: "38d7cf55-6057-4806-b438-8adcbd7e767b"). InnerVolumeSpecName "kube-api-access-45mwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:29.184940 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.184916 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle" pod "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" (UID: "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.185762 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.185734 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle" pod "638b03b2-d294-45c2-9bb2-a6e169eb0082" (UID: "638b03b2-d294-45c2-9bb2-a6e169eb0082"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.187369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.187335 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd" pod "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" (UID: "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c"). InnerVolumeSpecName "kube-api-access-4g7wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:29.189183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.189159 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util" pod "38d7cf55-6057-4806-b438-8adcbd7e767b" (UID: "38d7cf55-6057-4806-b438-8adcbd7e767b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.190158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.190133 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k" pod "6dbe982a-7cfb-4c2f-935d-5a625a6e5044" (UID: "6dbe982a-7cfb-4c2f-935d-5a625a6e5044"). InnerVolumeSpecName "kube-api-access-tqn7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:29.190279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.190255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util" pod "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" (UID: "31c2cd72-41b8-4a4e-a217-4d47a35e5f3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.191053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.191033 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util" pod "6dbe982a-7cfb-4c2f-935d-5a625a6e5044" (UID: "6dbe982a-7cfb-4c2f-935d-5a625a6e5044"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.191571 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.191551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util" pod "638b03b2-d294-45c2-9bb2-a6e169eb0082" (UID: "638b03b2-d294-45c2-9bb2-a6e169eb0082"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:29.281948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281916 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktz6j\" (UniqueName: \"kubernetes.io/projected/638b03b2-d294-45c2-9bb2-a6e169eb0082-kube-api-access-ktz6j\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.281948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281946 5051 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.281948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281958 5051 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-util\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281966 5051 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-util\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281975 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mwc\" (UniqueName: \"kubernetes.io/projected/38d7cf55-6057-4806-b438-8adcbd7e767b-kube-api-access-45mwc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.281983 5051 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-util\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282015 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7wd\" (UniqueName: \"kubernetes.io/projected/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-kube-api-access-4g7wd\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282024 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqn7k\" (UniqueName: \"kubernetes.io/projected/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-kube-api-access-tqn7k\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282033 5051 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38d7cf55-6057-4806-b438-8adcbd7e767b-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282041 5051 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31c2cd72-41b8-4a4e-a217-4d47a35e5f3c-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282048 5051 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/638b03b2-d294-45c2-9bb2-a6e169eb0082-util\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.282218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.282057 5051 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dbe982a-7cfb-4c2f-935d-5a625a6e5044-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:29.911964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.911924 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" Nov 03 22:27:29.911964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.911934 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wv6xz" event={"ID":"31c2cd72-41b8-4a4e-a217-4d47a35e5f3c","Type":"ContainerDied","Data":"89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c"} Nov 03 22:27:29.912221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.912015 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ede6c806a3ad6475d7a9074fca11b53a507da0a4187419b8a06c7b7e8c3c9c" Nov 03 22:27:29.913966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.913942 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" Nov 03 22:27:29.914103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.913970 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trsf9" event={"ID":"6dbe982a-7cfb-4c2f-935d-5a625a6e5044","Type":"ContainerDied","Data":"0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb"} Nov 03 22:27:29.914103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.914033 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfc179253e2504208ef21af9c49634c1310c604eb3b3c8ada36de85da0a9cfb" Nov 03 22:27:29.915713 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.915690 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" event={"ID":"38d7cf55-6057-4806-b438-8adcbd7e767b","Type":"ContainerDied","Data":"650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826"} Nov 03 22:27:29.915816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.915732 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650ea058e3e38510b72ef371c06ef4e1d20b3f351f9108ed5c85ec66b086e826" Nov 03 22:27:29.915816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.915802 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjjpqf" Nov 03 22:27:29.919184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.919163 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" event={"ID":"638b03b2-d294-45c2-9bb2-a6e169eb0082","Type":"ContainerDied","Data":"d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703"} Nov 03 22:27:29.919279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.919203 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d25f0b188055cdc81c8821ba4db73cfb2c772f1446e21a9bd6259cd16b02d703" Nov 03 22:27:29.919279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:29.919211 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akls7f" Nov 03 22:27:38.367864 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.367826 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z"] Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368344 5051 cpu_manager.go:414] "containerMap: removing container" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368365 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368384 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368392 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368406 5051 cpu_manager.go:414] "containerMap: removing container" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368413 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368426 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368433 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368442 5051 cpu_manager.go:414] "containerMap: removing container" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368450 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="util" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368478 5051 cpu_manager.go:414] "containerMap: removing container" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368484 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="pull" Nov 03 22:27:38.368489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368495 5051 cpu_manager.go:414] "containerMap: removing container" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="pull" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368502 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="pull" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368509 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368515 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368525 5051 cpu_manager.go:414] "containerMap: removing container" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368532 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368543 5051 cpu_manager.go:414] "containerMap: removing container" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368549 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368562 5051 cpu_manager.go:414] "containerMap: removing container" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="util" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368569 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="util" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368588 5051 cpu_manager.go:414] "containerMap: removing container" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368595 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368685 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="638b03b2-d294-45c2-9bb2-a6e169eb0082" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368696 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbe982a-7cfb-4c2f-935d-5a625a6e5044" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368706 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c2cd72-41b8-4a4e-a217-4d47a35e5f3c" containerName="extract" Nov 03 22:27:38.369155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.368716 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d7cf55-6057-4806-b438-8adcbd7e767b" containerName="extract" Nov 03 22:27:38.371685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.371624 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.447567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.447539 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:27:38.460907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.460868 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.460907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.460918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.461107 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.461026 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.461107 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.461051 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2g2f\" (UniqueName: \"kubernetes.io/projected/b46ea9a9-4b7a-497f-8e20-811f907f4076-kube-api-access-b2g2f\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.461107 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.461083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.461201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.461172 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.473960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.473939 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:27:38.474106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.474061 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:27:38.474310 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.474295 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:27:38.474371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.474334 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6hr8v\"" Nov 03 22:27:38.474426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.474373 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:27:38.476827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.476808 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z"] Nov 03 22:27:38.561753 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.561720 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.561943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.561762 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.561943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.561823 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.561943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.561847 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2g2f\" (UniqueName: \"kubernetes.io/projected/b46ea9a9-4b7a-497f-8e20-811f907f4076-kube-api-access-b2g2f\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.561943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.561886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.562221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.562075 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.562390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.562368 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.564401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.564381 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46ea9a9-4b7a-497f-8e20-811f907f4076-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.564708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.564688 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.564783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.564764 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.564840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.564781 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b46ea9a9-4b7a-497f-8e20-811f907f4076-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.623090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.623018 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2g2f\" (UniqueName: \"kubernetes.io/projected/b46ea9a9-4b7a-497f-8e20-811f907f4076-kube-api-access-b2g2f\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-rv89z\" (UID: \"b46ea9a9-4b7a-497f-8e20-811f907f4076\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:38.685085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:38.685048 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:39.185163 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:39.185009 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z"] Nov 03 22:27:39.188120 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:39.188095 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46ea9a9_4b7a_497f_8e20_811f907f4076.slice/crio-2df1bc5b4294c3e0faf727ea6ef9cf027016579507fd0ce84736cf8a6f922165 WatchSource:0}: Error finding container 2df1bc5b4294c3e0faf727ea6ef9cf027016579507fd0ce84736cf8a6f922165: Status 404 returned error can't find the container with id 2df1bc5b4294c3e0faf727ea6ef9cf027016579507fd0ce84736cf8a6f922165 Nov 03 22:27:39.962911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:39.962871 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" event={"ID":"b46ea9a9-4b7a-497f-8e20-811f907f4076","Type":"ContainerStarted","Data":"2df1bc5b4294c3e0faf727ea6ef9cf027016579507fd0ce84736cf8a6f922165"} Nov 03 22:27:42.979313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:42.979277 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" event={"ID":"b46ea9a9-4b7a-497f-8e20-811f907f4076","Type":"ContainerStarted","Data":"e58e4f6cea9bf42021f72bc78a202b3031ab7fa2f89c468c1927a67604742511"} Nov 03 22:27:43.985598 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:43.985563 5051 generic.go:356] "Generic (PLEG): container finished" podID="b46ea9a9-4b7a-497f-8e20-811f907f4076" containerID="e58e4f6cea9bf42021f72bc78a202b3031ab7fa2f89c468c1927a67604742511" exitCode=0 Nov 03 22:27:43.985969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:43.985605 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" event={"ID":"b46ea9a9-4b7a-497f-8e20-811f907f4076","Type":"ContainerDied","Data":"e58e4f6cea9bf42021f72bc78a202b3031ab7fa2f89c468c1927a67604742511"} Nov 03 22:27:44.176388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.176353 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t27g8"] Nov 03 22:27:44.178758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.178739 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:44.236542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.236474 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:27:44.240926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.240907 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wnk8h\"" Nov 03 22:27:44.295037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.295004 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t27g8"] Nov 03 22:27:44.313416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.313386 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmrh\" (UniqueName: \"kubernetes.io/projected/ecd47f33-4a76-447c-b457-17597360a16a-kube-api-access-zjmrh\") pod \"authorino-operator-7b7bb4bc77-t27g8\" (UID: \"ecd47f33-4a76-447c-b457-17597360a16a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:44.336137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.336109 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:27:44.414497 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.414463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmrh\" (UniqueName: \"kubernetes.io/projected/ecd47f33-4a76-447c-b457-17597360a16a-kube-api-access-zjmrh\") pod \"authorino-operator-7b7bb4bc77-t27g8\" (UID: \"ecd47f33-4a76-447c-b457-17597360a16a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:44.479074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.479041 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmrh\" (UniqueName: \"kubernetes.io/projected/ecd47f33-4a76-447c-b457-17597360a16a-kube-api-access-zjmrh\") pod \"authorino-operator-7b7bb4bc77-t27g8\" (UID: \"ecd47f33-4a76-447c-b457-17597360a16a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:44.493868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.493803 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:44.864931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.864902 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t27g8"] Nov 03 22:27:44.867936 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:44.867908 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd47f33_4a76_447c_b457_17597360a16a.slice/crio-c051047bdf4f4b31108873a7391628260214bcc7afef4563232f23f9a4e08696 WatchSource:0}: Error finding container c051047bdf4f4b31108873a7391628260214bcc7afef4563232f23f9a4e08696: Status 404 returned error can't find the container with id c051047bdf4f4b31108873a7391628260214bcc7afef4563232f23f9a4e08696 Nov 03 22:27:44.993090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.992978 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" event={"ID":"ecd47f33-4a76-447c-b457-17597360a16a","Type":"ContainerStarted","Data":"c051047bdf4f4b31108873a7391628260214bcc7afef4563232f23f9a4e08696"} Nov 03 22:27:44.995414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.995386 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" event={"ID":"b46ea9a9-4b7a-497f-8e20-811f907f4076","Type":"ContainerStarted","Data":"cceb49c5e9a406c166fe6308921670b0a4afe2896d03792cd4598641f1c3efe6"} Nov 03 22:27:44.995573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:44.995557 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:45.164326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:45.164272 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" podStartSLOduration=2.545119776 podStartE2EDuration="7.164256473s" totalImagesPullingTime="4.161134784s" totalInitContainerRuntime="458.001913ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:45.162262085 +0000 UTC m=+338.899606309" watchObservedRunningTime="2026-11-03 22:27:45.164256473 +0000 UTC m=+338.901600698" Nov 03 22:27:49.017305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:49.017271 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" event={"ID":"ecd47f33-4a76-447c-b457-17597360a16a","Type":"ContainerStarted","Data":"5068e4d7442300a12a16afec2ddf40bac23a9a1cf59c08226740551e853797ca"} Nov 03 22:27:49.017749 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:49.017355 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:27:49.135809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:49.135756 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" podStartSLOduration=1.835142295 podStartE2EDuration="5.135742919s" totalImagesPullingTime="3.300600624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:49.134122719 +0000 UTC m=+342.871466956" watchObservedRunningTime="2026-11-03 22:27:49.135742919 +0000 UTC m=+342.873087156" Nov 03 22:27:51.621627 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:27:51.621592 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:27:52.816271 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.816236 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 22:27:52.819850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.819830 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:52.872152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.872027 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 22:27:52.891447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.891397 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6t2gp\"" Nov 03 22:27:52.892961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.892923 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgd7c\" (UniqueName: \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:52.893178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.893068 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:52.993823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.993791 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgd7c\" (UniqueName: \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:52.994034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.993833 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:52.994291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:52.994273 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:53.059915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:53.059873 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgd7c\" (UniqueName: \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\") pod \"kuadrant-operator-controller-manager-678d87766f-pzscd\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:53.131677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:53.131632 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:27:53.560102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:53.560020 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 22:27:53.563251 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:27:53.563224 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107cdd6b_efe1_4d9e_8646_ed386019fea2.slice/crio-391edae764f8905bf72556f066f855ed7ecf6175f01e8247d2a4803de170a7c0 WatchSource:0}: Error finding container 391edae764f8905bf72556f066f855ed7ecf6175f01e8247d2a4803de170a7c0: Status 404 returned error can't find the container with id 391edae764f8905bf72556f066f855ed7ecf6175f01e8247d2a4803de170a7c0 Nov 03 22:27:54.041427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:54.041392 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" event={"ID":"107cdd6b-efe1-4d9e-8646-ed386019fea2","Type":"ContainerStarted","Data":"391edae764f8905bf72556f066f855ed7ecf6175f01e8247d2a4803de170a7c0"} Nov 03 22:27:56.004284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:56.004255 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-rv89z" Nov 03 22:27:59.066454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:59.066416 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" event={"ID":"107cdd6b-efe1-4d9e-8646-ed386019fea2","Type":"ContainerStarted","Data":"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062"} Nov 03 22:27:59.067024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:27:59.066547 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:28:00.025482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:00.025445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t27g8" Nov 03 22:28:00.158441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:00.158380 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" podStartSLOduration=3.621408159 podStartE2EDuration="8.158366109s" totalImagesPullingTime="4.53695795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:59.287614913 +0000 UTC m=+353.024959137" watchObservedRunningTime="2026-11-03 22:28:00.158366109 +0000 UTC m=+353.895710345" Nov 03 22:28:09.355924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.355886 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:09.360842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.360818 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.417384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.417354 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-plwfc\"" Nov 03 22:28:09.419906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.419878 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:28:09.422967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.422936 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:09.529589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.529544 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.529761 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.529711 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5vt\" (UniqueName: \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.608483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.608404 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:09.631218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.631178 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5vt\" (UniqueName: \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.631620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.631270 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.632264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.632236 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.711923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.711886 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5vt\" (UniqueName: \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\") pod \"limitador-limitador-74dfcfb894-vr4nf\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:09.973211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:09.973135 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:10.035113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.032462 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh"] Nov 03 22:28:10.037581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.036333 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.074536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.074509 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 22:28:10.098243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.098213 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh"] Nov 03 22:28:10.134667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.134640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.134848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.134679 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.134848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.134698 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.134848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.134757 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7p9c\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-kube-api-access-l7p9c\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.135011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.134934 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d71f6e-e71d-4197-b99b-327870fb690c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.135052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.135011 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.135052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.135037 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.135124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.135108 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.135158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.135147 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.162178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.162155 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lp6gv\"" Nov 03 22:28:10.236124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236040 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7p9c\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-kube-api-access-l7p9c\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236125 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d71f6e-e71d-4197-b99b-327870fb690c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236152 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236169 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236222 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236257 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236303 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236688 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236747 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236774 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.236928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236907 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d71f6e-e71d-4197-b99b-327870fb690c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.237037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.236968 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.238627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.238607 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.238841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.238821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.309175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.309145 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7p9c\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-kube-api-access-l7p9c\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.312901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.312869 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d71f6e-e71d-4197-b99b-327870fb690c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh\" (UID: \"d6d71f6e-e71d-4197-b99b-327870fb690c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.352779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.352739 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:10.363303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.358854 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:10.367760 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:10.367688 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86ca348_9ded_41c7_969c_32db55438ac1.slice/crio-5a39438eb0b7b0c60d2acf426120776611980322cdc117ec87b2e14989358e19 WatchSource:0}: Error finding container 5a39438eb0b7b0c60d2acf426120776611980322cdc117ec87b2e14989358e19: Status 404 returned error can't find the container with id 5a39438eb0b7b0c60d2acf426120776611980322cdc117ec87b2e14989358e19 Nov 03 22:28:10.744779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.744746 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh"] Nov 03 22:28:10.748713 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:10.748682 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d71f6e_e71d_4197_b99b_327870fb690c.slice/crio-946fd159becf4992fb1dbc11d1d4939b4f5ed962fc8866e736784c40274a90f5 WatchSource:0}: Error finding container 946fd159becf4992fb1dbc11d1d4939b4f5ed962fc8866e736784c40274a90f5: Status 404 returned error can't find the container with id 946fd159becf4992fb1dbc11d1d4939b4f5ed962fc8866e736784c40274a90f5 Nov 03 22:28:10.751053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.751024 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:28:10.751123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.751091 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:28:10.751169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:10.751123 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:28:11.125504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.125462 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" event={"ID":"d6d71f6e-e71d-4197-b99b-327870fb690c","Type":"ContainerStarted","Data":"333d5cfd05d377c9e7e57a72cc27b778ff3ca8e40ed3305747d28af83bb319d4"} Nov 03 22:28:11.125724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.125530 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" event={"ID":"d6d71f6e-e71d-4197-b99b-327870fb690c","Type":"ContainerStarted","Data":"946fd159becf4992fb1dbc11d1d4939b4f5ed962fc8866e736784c40274a90f5"} Nov 03 22:28:11.126755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.126731 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" event={"ID":"d86ca348-9ded-41c7-969c-32db55438ac1","Type":"ContainerStarted","Data":"5a39438eb0b7b0c60d2acf426120776611980322cdc117ec87b2e14989358e19"} Nov 03 22:28:11.258462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.258406 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" podStartSLOduration=2.258390891 podStartE2EDuration="2.258390891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:11.257186002 +0000 UTC m=+364.994530227" watchObservedRunningTime="2026-11-03 22:28:11.258390891 +0000 UTC m=+364.995735115" Nov 03 22:28:11.293021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.292970 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:11.301084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.301042 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:11.345212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.345179 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdxq\" (UniqueName: \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\") pod \"authorino-57f7f8659f-lzwdq\" (UID: \"1674ad48-13fd-41b6-b05d-84597f02d8db\") " pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:11.353118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.353093 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:11.358119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.358098 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:11.363832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.363805 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:11.367217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.367186 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2nn5\"" Nov 03 22:28:11.445968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.445890 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdxq\" (UniqueName: \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\") pod \"authorino-57f7f8659f-lzwdq\" (UID: \"1674ad48-13fd-41b6-b05d-84597f02d8db\") " pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:11.514374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.514343 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdxq\" (UniqueName: \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\") pod \"authorino-57f7f8659f-lzwdq\" (UID: \"1674ad48-13fd-41b6-b05d-84597f02d8db\") " pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:11.631943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:11.631905 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:12.005066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:12.005035 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:12.008662 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:12.008627 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1674ad48_13fd_41b6_b05d_84597f02d8db.slice/crio-52d436ddf5cfbd462fce68ce1b7e0a5135ad6deb2be339a457fca8ea2d4b7bfb WatchSource:0}: Error finding container 52d436ddf5cfbd462fce68ce1b7e0a5135ad6deb2be339a457fca8ea2d4b7bfb: Status 404 returned error can't find the container with id 52d436ddf5cfbd462fce68ce1b7e0a5135ad6deb2be339a457fca8ea2d4b7bfb Nov 03 22:28:12.133588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:12.133546 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" event={"ID":"1674ad48-13fd-41b6-b05d-84597f02d8db","Type":"ContainerStarted","Data":"52d436ddf5cfbd462fce68ce1b7e0a5135ad6deb2be339a457fca8ea2d4b7bfb"} Nov 03 22:28:12.134208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:12.134180 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:12.135196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:12.135172 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh" Nov 03 22:28:13.139250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:13.139217 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" event={"ID":"d86ca348-9ded-41c7-969c-32db55438ac1","Type":"ContainerStarted","Data":"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851"} Nov 03 22:28:13.139696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:13.139508 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:13.256636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:13.256580 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" podStartSLOduration=2.317076524 podStartE2EDuration="4.256564965s" totalImagesPullingTime="1.939488441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:13.255111101 +0000 UTC m=+366.992455327" watchObservedRunningTime="2026-11-03 22:28:13.256564965 +0000 UTC m=+366.993909198" Nov 03 22:28:15.150867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:15.150833 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" event={"ID":"1674ad48-13fd-41b6-b05d-84597f02d8db","Type":"ContainerStarted","Data":"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f"} Nov 03 22:28:15.266151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:15.266092 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" podStartSLOduration=1.720077117 podStartE2EDuration="4.266071991s" totalImagesPullingTime="2.545994874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:15.264800841 +0000 UTC m=+369.002145070" watchObservedRunningTime="2026-11-03 22:28:15.266071991 +0000 UTC m=+369.003416215" Nov 03 22:28:17.429257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.429221 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:17.429649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.429423 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" podUID="1674ad48-13fd-41b6-b05d-84597f02d8db" containerName="authorino" containerID="cri-o://c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f" gracePeriod=30 Nov 03 22:28:17.722701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.722627 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:17.791745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.791711 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\" (UniqueName: \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\") pod \"1674ad48-13fd-41b6-b05d-84597f02d8db\" (UID: \"1674ad48-13fd-41b6-b05d-84597f02d8db\") " Nov 03 22:28:17.794072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.794045 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq" pod "1674ad48-13fd-41b6-b05d-84597f02d8db" (UID: "1674ad48-13fd-41b6-b05d-84597f02d8db"). InnerVolumeSpecName "kube-api-access-kvdxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:17.892373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:17.892336 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdxq\" (UniqueName: \"kubernetes.io/projected/1674ad48-13fd-41b6-b05d-84597f02d8db-kube-api-access-kvdxq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:28:18.166745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.166708 5051 generic.go:356] "Generic (PLEG): container finished" podID="1674ad48-13fd-41b6-b05d-84597f02d8db" containerID="c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f" exitCode=0 Nov 03 22:28:18.166927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.166754 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" Nov 03 22:28:18.166927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.166769 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" event={"ID":"1674ad48-13fd-41b6-b05d-84597f02d8db","Type":"ContainerDied","Data":"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f"} Nov 03 22:28:18.166927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.166808 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lzwdq" event={"ID":"1674ad48-13fd-41b6-b05d-84597f02d8db","Type":"ContainerDied","Data":"52d436ddf5cfbd462fce68ce1b7e0a5135ad6deb2be339a457fca8ea2d4b7bfb"} Nov 03 22:28:18.166927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.166829 5051 scope.go:122] "RemoveContainer" containerID="c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f" Nov 03 22:28:18.178512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.178492 5051 scope.go:122] "RemoveContainer" containerID="c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f" Nov 03 22:28:18.178928 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:28:18.178900 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f\": container with ID starting with c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f not found: ID does not exist" containerID="c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f" Nov 03 22:28:18.179072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.178940 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f"} err="failed to get container status \"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f\": rpc error: code = NotFound desc = could not find container \"c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f\": container with ID starting with c76951c7d6182c264a6e087dbe81a429216b1249ab72fb660a4a3855d277ee3f not found: ID does not exist" Nov 03 22:28:18.328856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.328820 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:18.334375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.334353 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lzwdq"] Nov 03 22:28:18.405805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:18.405777 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1674ad48-13fd-41b6-b05d-84597f02d8db" path="/var/lib/kubelet/pods/1674ad48-13fd-41b6-b05d-84597f02d8db/volumes" Nov 03 22:28:21.621879 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:28:21.621843 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:28:22.105908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.105876 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64"] Nov 03 22:28:22.106267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.106253 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1674ad48-13fd-41b6-b05d-84597f02d8db" containerName="authorino" Nov 03 22:28:22.106315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.106269 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1674ad48-13fd-41b6-b05d-84597f02d8db" containerName="authorino" Nov 03 22:28:22.106356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.106347 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1674ad48-13fd-41b6-b05d-84597f02d8db" containerName="authorino" Nov 03 22:28:22.109255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.109239 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.125975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.125948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.126120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.126069 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/743264a8-12f2-4ae4-9499-17f03b60978d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.126120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.126099 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.126191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.126154 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96l9p\" (UniqueName: \"kubernetes.io/projected/743264a8-12f2-4ae4-9499-17f03b60978d-kube-api-access-96l9p\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.161846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.161814 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:28:22.211316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.211282 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:28:22.211316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.211296 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-7sn62\"" Nov 03 22:28:22.227304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227278 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96l9p\" (UniqueName: \"kubernetes.io/projected/743264a8-12f2-4ae4-9499-17f03b60978d-kube-api-access-96l9p\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.227409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227334 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.227409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227383 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/743264a8-12f2-4ae4-9499-17f03b60978d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.227409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227403 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.227709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.227774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.227757 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/743264a8-12f2-4ae4-9499-17f03b60978d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.311338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.311252 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 22:28:22.313197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.313176 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64"] Nov 03 22:28:22.314206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.314180 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96l9p\" (UniqueName: \"kubernetes.io/projected/743264a8-12f2-4ae4-9499-17f03b60978d-kube-api-access-96l9p\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.318290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.318265 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/743264a8-12f2-4ae4-9499-17f03b60978d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-l4w64\" (UID: \"743264a8-12f2-4ae4-9499-17f03b60978d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.420802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.420722 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:22.844761 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:22.844670 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64"] Nov 03 22:28:22.847721 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:22.847693 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743264a8_12f2_4ae4_9499_17f03b60978d.slice/crio-f13b6ec3d11a6a02de39548716605f04163d5ea0125e8894fcfa8a0bc1af8755 WatchSource:0}: Error finding container f13b6ec3d11a6a02de39548716605f04163d5ea0125e8894fcfa8a0bc1af8755: Status 404 returned error can't find the container with id f13b6ec3d11a6a02de39548716605f04163d5ea0125e8894fcfa8a0bc1af8755 Nov 03 22:28:23.192245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:23.192209 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" event={"ID":"743264a8-12f2-4ae4-9499-17f03b60978d","Type":"ContainerStarted","Data":"f13b6ec3d11a6a02de39548716605f04163d5ea0125e8894fcfa8a0bc1af8755"} Nov 03 22:28:24.145155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:24.145124 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:26.210445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:26.210402 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" event={"ID":"743264a8-12f2-4ae4-9499-17f03b60978d","Type":"ContainerStarted","Data":"9a2dbc58b17b8d3fa3879dff2bb4e8ac8b962821d29b7e7772ed10640046cfdb"} Nov 03 22:28:26.210846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:26.210458 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:26.379596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:26.379541 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" podStartSLOduration=1.839747609 podStartE2EDuration="4.379526713s" totalImagesPullingTime="2.539779104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:26.378636594 +0000 UTC m=+380.115980818" watchObservedRunningTime="2026-11-03 22:28:26.379526713 +0000 UTC m=+380.116870938" Nov 03 22:28:29.169352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:29.169306 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:29.170275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:29.169590 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" podUID="d86ca348-9ded-41c7-969c-32db55438ac1" containerName="limitador" containerID="cri-o://9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851" gracePeriod=30 Nov 03 22:28:30.129498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.129471 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:30.198955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.198918 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\" (UniqueName: \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\") pod \"d86ca348-9ded-41c7-969c-32db55438ac1\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " Nov 03 22:28:30.199461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.199028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\" (UniqueName: \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\") pod \"d86ca348-9ded-41c7-969c-32db55438ac1\" (UID: \"d86ca348-9ded-41c7-969c-32db55438ac1\") " Nov 03 22:28:30.199461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.199305 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file" pod "d86ca348-9ded-41c7-969c-32db55438ac1" (UID: "d86ca348-9ded-41c7-969c-32db55438ac1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:30.201369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.201342 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt" pod "d86ca348-9ded-41c7-969c-32db55438ac1" (UID: "d86ca348-9ded-41c7-969c-32db55438ac1"). InnerVolumeSpecName "kube-api-access-kv5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:30.235674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.235641 5051 generic.go:356] "Generic (PLEG): container finished" podID="d86ca348-9ded-41c7-969c-32db55438ac1" containerID="9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851" exitCode=0 Nov 03 22:28:30.235813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.235689 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" Nov 03 22:28:30.235813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.235709 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" event={"ID":"d86ca348-9ded-41c7-969c-32db55438ac1","Type":"ContainerDied","Data":"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851"} Nov 03 22:28:30.235813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.235750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vr4nf" event={"ID":"d86ca348-9ded-41c7-969c-32db55438ac1","Type":"ContainerDied","Data":"5a39438eb0b7b0c60d2acf426120776611980322cdc117ec87b2e14989358e19"} Nov 03 22:28:30.235813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.235771 5051 scope.go:122] "RemoveContainer" containerID="9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851" Nov 03 22:28:30.245082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.245057 5051 scope.go:122] "RemoveContainer" containerID="9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851" Nov 03 22:28:30.245597 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:28:30.245558 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851\": container with ID starting with 9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851 not found: ID does not exist" containerID="9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851" Nov 03 22:28:30.245705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.245613 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851"} err="failed to get container status \"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851\": rpc error: code = NotFound desc = could not find container \"9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851\": container with ID starting with 9309331e7f554cba70e437b93915c284760192a2c0cebae7d313801e2775a851 not found: ID does not exist" Nov 03 22:28:30.300474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.300440 5051 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86ca348-9ded-41c7-969c-32db55438ac1-config-file\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:28:30.300474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.300468 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv5vt\" (UniqueName: \"kubernetes.io/projected/d86ca348-9ded-41c7-969c-32db55438ac1-kube-api-access-kv5vt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:28:30.356152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.356118 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:30.361058 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.361028 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vr4nf"] Nov 03 22:28:30.405933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:30.405903 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86ca348-9ded-41c7-969c-32db55438ac1" path="/var/lib/kubelet/pods/d86ca348-9ded-41c7-969c-32db55438ac1/volumes" Nov 03 22:28:35.643174 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.643139 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88"] Nov 03 22:28:35.643567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.643504 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d86ca348-9ded-41c7-969c-32db55438ac1" containerName="limitador" Nov 03 22:28:35.643567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.643516 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86ca348-9ded-41c7-969c-32db55438ac1" containerName="limitador" Nov 03 22:28:35.643636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.643597 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86ca348-9ded-41c7-969c-32db55438ac1" containerName="limitador" Nov 03 22:28:35.646007 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.645968 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.698278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.698251 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 22:28:35.701881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.701854 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:35.747571 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.747546 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:35.747724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.747578 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1091c07f-30cf-4e09-a7a3-06a236aa4c59-data\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.747724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.747616 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x742x\" (UniqueName: \"kubernetes.io/projected/1091c07f-30cf-4e09-a7a3-06a236aa4c59-kube-api-access-x742x\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.747849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.747779 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-glmf9\"" Nov 03 22:28:35.749658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.749632 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88"] Nov 03 22:28:35.848220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.848178 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1091c07f-30cf-4e09-a7a3-06a236aa4c59-data\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.848400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.848239 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x742x\" (UniqueName: \"kubernetes.io/projected/1091c07f-30cf-4e09-a7a3-06a236aa4c59-kube-api-access-x742x\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.848570 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.848552 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1091c07f-30cf-4e09-a7a3-06a236aa4c59-data\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.916271 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.916198 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x742x\" (UniqueName: \"kubernetes.io/projected/1091c07f-30cf-4e09-a7a3-06a236aa4c59-kube-api-access-x742x\") pod \"postgres-7cdbf6dc44-r7z88\" (UID: \"1091c07f-30cf-4e09-a7a3-06a236aa4c59\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:35.956787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:35.956753 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:36.330129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:36.329981 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88"] Nov 03 22:28:36.332873 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:36.332846 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1091c07f_30cf_4e09_a7a3_06a236aa4c59.slice/crio-3963fa5e39954403b51ee1a4ab3edf4cdedc6a376e429ed3b6641688c29843c9 WatchSource:0}: Error finding container 3963fa5e39954403b51ee1a4ab3edf4cdedc6a376e429ed3b6641688c29843c9: Status 404 returned error can't find the container with id 3963fa5e39954403b51ee1a4ab3edf4cdedc6a376e429ed3b6641688c29843c9 Nov 03 22:28:37.276142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:37.276106 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" event={"ID":"1091c07f-30cf-4e09-a7a3-06a236aa4c59","Type":"ContainerStarted","Data":"3963fa5e39954403b51ee1a4ab3edf4cdedc6a376e429ed3b6641688c29843c9"} Nov 03 22:28:38.222422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.222387 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95"] Nov 03 22:28:38.225387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.225370 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.270981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.270944 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-cert\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.271162 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.271083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4755e08f-5c69-4416-a51a-c2c74a3e6655-manager-config\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.271162 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.271130 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-metrics-certs\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.271235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.271158 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/4755e08f-5c69-4416-a51a-c2c74a3e6655-kube-api-access-fmfsn\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.327634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.327601 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 22:28:38.328074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.327712 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 22:28:38.328074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.327816 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-q2qdk\"" Nov 03 22:28:38.328074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.327880 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:28:38.328074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.327884 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 22:28:38.328398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.328383 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:28:38.330388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.330364 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95"] Nov 03 22:28:38.372140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.372117 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4755e08f-5c69-4416-a51a-c2c74a3e6655-manager-config\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.372275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.372159 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-metrics-certs\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.372275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.372187 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/4755e08f-5c69-4416-a51a-c2c74a3e6655-kube-api-access-fmfsn\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.372275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.372216 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-cert\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.372860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.372834 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4755e08f-5c69-4416-a51a-c2c74a3e6655-manager-config\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.375017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.374968 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-metrics-certs\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.375123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.375105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4755e08f-5c69-4416-a51a-c2c74a3e6655-cert\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.433283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.433258 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/4755e08f-5c69-4416-a51a-c2c74a3e6655-kube-api-access-fmfsn\") pod \"jobset-controller-manager-7f469b8c9b-6nd95\" (UID: \"4755e08f-5c69-4416-a51a-c2c74a3e6655\") " pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.536737 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.536647 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:38.927612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:38.926494 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95"] Nov 03 22:28:38.933131 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:28:38.933099 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4755e08f_5c69_4416_a51a_c2c74a3e6655.slice/crio-ff71bf4c87b989378596216747da875c757dd0d1fdf4e8d4710f10d8a4804e06 WatchSource:0}: Error finding container ff71bf4c87b989378596216747da875c757dd0d1fdf4e8d4710f10d8a4804e06: Status 404 returned error can't find the container with id ff71bf4c87b989378596216747da875c757dd0d1fdf4e8d4710f10d8a4804e06 Nov 03 22:28:39.288687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:39.288599 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" event={"ID":"4755e08f-5c69-4416-a51a-c2c74a3e6655","Type":"ContainerStarted","Data":"ff71bf4c87b989378596216747da875c757dd0d1fdf4e8d4710f10d8a4804e06"} Nov 03 22:28:43.321323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:43.321288 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" event={"ID":"1091c07f-30cf-4e09-a7a3-06a236aa4c59","Type":"ContainerStarted","Data":"cdc2c6f72106924dded6a417fcee15ebdc1bd0ab3abba9398c0992a846b1707a"} Nov 03 22:28:43.321740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:43.321429 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:43.438052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:43.437983 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" podStartSLOduration=2.035153062 podStartE2EDuration="8.437969418s" totalImagesPullingTime="6.402816356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:43.436426834 +0000 UTC m=+397.173771059" watchObservedRunningTime="2026-11-03 22:28:43.437969418 +0000 UTC m=+397.175313642" Nov 03 22:28:47.217605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:47.217512 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-l4w64" Nov 03 22:28:48.347447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:48.347406 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" event={"ID":"4755e08f-5c69-4416-a51a-c2c74a3e6655","Type":"ContainerStarted","Data":"a8cc4b07dd096828532a8a3e9dc6832b6dff1f5f437f417813ecde16cbed98d2"} Nov 03 22:28:48.347837 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:48.347554 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:28:48.464270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:48.464212 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" podStartSLOduration=1.6498161009999999 podStartE2EDuration="10.464196562s" totalImagesPullingTime="8.814380461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:48.462336141 +0000 UTC m=+402.199680367" watchObservedRunningTime="2026-11-03 22:28:48.464196562 +0000 UTC m=+402.201540857" Nov 03 22:28:49.356567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:49.356539 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r7z88" Nov 03 22:28:51.621748 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:28:51.621712 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:28:59.359247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:28:59.359217 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7f469b8c9b-6nd95" Nov 03 22:29:21.622509 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:29:21.622471 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:29:44.545281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.545243 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:44.548632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.548609 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:44.603337 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.603065 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2nn5\"" Nov 03 22:29:44.605305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.605280 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:44.647217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.647182 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9b7r\" (UniqueName: \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\") pod \"authorino-6f6944bcdd-dlhbc\" (UID: \"c299215d-4efd-4a19-9550-ed59496f9325\") " pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:44.748130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.748100 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9b7r\" (UniqueName: \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\") pod \"authorino-6f6944bcdd-dlhbc\" (UID: \"c299215d-4efd-4a19-9550-ed59496f9325\") " pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:44.810402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.810285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9b7r\" (UniqueName: \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\") pod \"authorino-6f6944bcdd-dlhbc\" (UID: \"c299215d-4efd-4a19-9550-ed59496f9325\") " pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:44.877743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.877711 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:44.912194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.912155 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:44.943735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.942256 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:44.947455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:44.947434 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:45.003379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.003349 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:45.049751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.049723 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzll\" (UniqueName: \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\") pod \"authorino-669969f6b5-ntnd8\" (UID: \"95587d49-bc69-4eba-8fce-76d67fac3d74\") " pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:45.150741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.150716 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzll\" (UniqueName: \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\") pod \"authorino-669969f6b5-ntnd8\" (UID: \"95587d49-bc69-4eba-8fce-76d67fac3d74\") " pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:45.211564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.211532 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzll\" (UniqueName: \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\") pod \"authorino-669969f6b5-ntnd8\" (UID: \"95587d49-bc69-4eba-8fce-76d67fac3d74\") " pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:45.249138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.249110 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:45.251949 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:29:45.251924 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc299215d_4efd_4a19_9550_ed59496f9325.slice/crio-e65517cc299ab4fef0ee4d484e48e0e1e7c2be42c31c5e5ba4ef324454dee038 WatchSource:0}: Error finding container e65517cc299ab4fef0ee4d484e48e0e1e7c2be42c31c5e5ba4ef324454dee038: Status 404 returned error can't find the container with id e65517cc299ab4fef0ee4d484e48e0e1e7c2be42c31c5e5ba4ef324454dee038 Nov 03 22:29:45.260317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.260295 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:45.371976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.371946 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:45.624042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.623982 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" event={"ID":"c299215d-4efd-4a19-9550-ed59496f9325","Type":"ContainerStarted","Data":"e65517cc299ab4fef0ee4d484e48e0e1e7c2be42c31c5e5ba4ef324454dee038"} Nov 03 22:29:45.632738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:45.632698 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:45.635380 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:29:45.635341 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95587d49_bc69_4eba_8fce_76d67fac3d74.slice/crio-07fe7d492e206feada11f406374a1172953fab7694f437d70bf153268a925c07 WatchSource:0}: Error finding container 07fe7d492e206feada11f406374a1172953fab7694f437d70bf153268a925c07: Status 404 returned error can't find the container with id 07fe7d492e206feada11f406374a1172953fab7694f437d70bf153268a925c07 Nov 03 22:29:46.635170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.635123 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669969f6b5-ntnd8" event={"ID":"95587d49-bc69-4eba-8fce-76d67fac3d74","Type":"ContainerStarted","Data":"07fe7d492e206feada11f406374a1172953fab7694f437d70bf153268a925c07"} Nov 03 22:29:46.804458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.804422 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:29:46.808066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.808043 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.860147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.860120 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89xv7\"" Nov 03 22:29:46.865135 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865107 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865159 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865183 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865278 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865330 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865400 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865487 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqj5\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.865537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.865526 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.912562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.912493 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:29:46.966824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.966787 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.966890 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.966929 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.966944 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.966970 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967030 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967080 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967106 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967139 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqj5\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967443 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967725 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967827 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.967857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.967841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.968103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.968079 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.969393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.969377 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:46.969717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:46.969702 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:47.027197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.027167 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqj5\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:47.027296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.027207 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jr4dh\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:47.121287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.121268 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:47.555201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.555170 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:29:47.557420 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:29:47.557390 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6c0a15_936f_4ed4_b31b_02107af53d62.slice/crio-b575105ff281a5efbfab184ee08b26c33a737740b25f127556b918c4ff9b06e0 WatchSource:0}: Error finding container b575105ff281a5efbfab184ee08b26c33a737740b25f127556b918c4ff9b06e0: Status 404 returned error can't find the container with id b575105ff281a5efbfab184ee08b26c33a737740b25f127556b918c4ff9b06e0 Nov 03 22:29:47.559641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.559607 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:47.559740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.559686 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:47.559740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.559733 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:47.645691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.645656 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" event={"ID":"c299215d-4efd-4a19-9550-ed59496f9325","Type":"ContainerStarted","Data":"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751"} Nov 03 22:29:47.646197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.645713 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" podUID="c299215d-4efd-4a19-9550-ed59496f9325" containerName="authorino" containerID="cri-o://a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751" gracePeriod=30 Nov 03 22:29:47.647497 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.647465 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669969f6b5-ntnd8" event={"ID":"95587d49-bc69-4eba-8fce-76d67fac3d74","Type":"ContainerStarted","Data":"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3"} Nov 03 22:29:47.647645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.647547 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-669969f6b5-ntnd8" podUID="95587d49-bc69-4eba-8fce-76d67fac3d74" containerName="authorino" containerID="cri-o://d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3" gracePeriod=30 Nov 03 22:29:47.649565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.649541 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" event={"ID":"df6c0a15-936f-4ed4-b31b-02107af53d62","Type":"ContainerStarted","Data":"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7"} Nov 03 22:29:47.649715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.649695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" event={"ID":"df6c0a15-936f-4ed4-b31b-02107af53d62","Type":"ContainerStarted","Data":"b575105ff281a5efbfab184ee08b26c33a737740b25f127556b918c4ff9b06e0"} Nov 03 22:29:47.759393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.759293 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" podStartSLOduration=2.1806751699999998 podStartE2EDuration="3.759279354s" totalImagesPullingTime="1.578604184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:47.75782869 +0000 UTC m=+461.495172916" watchObservedRunningTime="2026-11-03 22:29:47.759279354 +0000 UTC m=+461.496623579" Nov 03 22:29:47.875220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.875143 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-669969f6b5-ntnd8" podStartSLOduration=2.445560416 podStartE2EDuration="3.875122684s" totalImagesPullingTime="1.429562268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:47.87060201 +0000 UTC m=+461.607946235" watchObservedRunningTime="2026-11-03 22:29:47.875122684 +0000 UTC m=+461.612466908" Nov 03 22:29:47.953937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.953904 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:47.957332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.957311 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:47.992744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:47.992678 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podStartSLOduration=1.992662858 podStartE2EDuration="1.992662858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:47.991589711 +0000 UTC m=+461.728934067" watchObservedRunningTime="2026-11-03 22:29:47.992662858 +0000 UTC m=+461.730007084" Nov 03 22:29:48.079081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.078323 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\" (UniqueName: \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\") pod \"c299215d-4efd-4a19-9550-ed59496f9325\" (UID: \"c299215d-4efd-4a19-9550-ed59496f9325\") " Nov 03 22:29:48.079081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.078383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\" (UniqueName: \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\") pod \"95587d49-bc69-4eba-8fce-76d67fac3d74\" (UID: \"95587d49-bc69-4eba-8fce-76d67fac3d74\") " Nov 03 22:29:48.083528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.083489 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll" pod "95587d49-bc69-4eba-8fce-76d67fac3d74" (UID: "95587d49-bc69-4eba-8fce-76d67fac3d74"). InnerVolumeSpecName "kube-api-access-zbzll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:48.084747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.084719 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r" pod "c299215d-4efd-4a19-9550-ed59496f9325" (UID: "c299215d-4efd-4a19-9550-ed59496f9325"). InnerVolumeSpecName "kube-api-access-s9b7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:48.121975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.121946 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:48.123370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.123341 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:48.123473 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.123381 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:48.181015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.180641 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzll\" (UniqueName: \"kubernetes.io/projected/95587d49-bc69-4eba-8fce-76d67fac3d74-kube-api-access-zbzll\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:48.181015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.180676 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9b7r\" (UniqueName: \"kubernetes.io/projected/c299215d-4efd-4a19-9550-ed59496f9325-kube-api-access-s9b7r\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:48.656199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.656164 5051 generic.go:356] "Generic (PLEG): container finished" podID="c299215d-4efd-4a19-9550-ed59496f9325" containerID="a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751" exitCode=0 Nov 03 22:29:48.656841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.656820 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" Nov 03 22:29:48.657476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.657442 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" event={"ID":"c299215d-4efd-4a19-9550-ed59496f9325","Type":"ContainerDied","Data":"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751"} Nov 03 22:29:48.657665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.657647 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-dlhbc" event={"ID":"c299215d-4efd-4a19-9550-ed59496f9325","Type":"ContainerDied","Data":"e65517cc299ab4fef0ee4d484e48e0e1e7c2be42c31c5e5ba4ef324454dee038"} Nov 03 22:29:48.657782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.657769 5051 scope.go:122] "RemoveContainer" containerID="a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751" Nov 03 22:29:48.673232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.669293 5051 generic.go:356] "Generic (PLEG): container finished" podID="95587d49-bc69-4eba-8fce-76d67fac3d74" containerID="d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3" exitCode=0 Nov 03 22:29:48.673435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.673413 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-669969f6b5-ntnd8" Nov 03 22:29:48.673688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.673592 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669969f6b5-ntnd8" event={"ID":"95587d49-bc69-4eba-8fce-76d67fac3d74","Type":"ContainerDied","Data":"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3"} Nov 03 22:29:48.673688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.673650 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-669969f6b5-ntnd8" event={"ID":"95587d49-bc69-4eba-8fce-76d67fac3d74","Type":"ContainerDied","Data":"07fe7d492e206feada11f406374a1172953fab7694f437d70bf153268a925c07"} Nov 03 22:29:48.688930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.688908 5051 scope.go:122] "RemoveContainer" containerID="a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751" Nov 03 22:29:48.689300 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:29:48.689279 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751\": container with ID starting with a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751 not found: ID does not exist" containerID="a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751" Nov 03 22:29:48.689369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.689310 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751"} err="failed to get container status \"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751\": rpc error: code = NotFound desc = could not find container \"a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751\": container with ID starting with a7aee18ee620adc04125c42973b52b036452b865fbc7221744c1188d666c8751 not found: ID does not exist" Nov 03 22:29:48.689369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.689327 5051 scope.go:122] "RemoveContainer" containerID="d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3" Nov 03 22:29:48.697879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.697857 5051 scope.go:122] "RemoveContainer" containerID="d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3" Nov 03 22:29:48.698176 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:29:48.698158 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3\": container with ID starting with d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3 not found: ID does not exist" containerID="d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3" Nov 03 22:29:48.698227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.698184 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3"} err="failed to get container status \"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3\": rpc error: code = NotFound desc = could not find container \"d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3\": container with ID starting with d4d8b293fe67af8b221c46472888ba02badff7450ffa92271719ed58c6bbaba3 not found: ID does not exist" Nov 03 22:29:48.836581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.836547 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:48.840504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.840476 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-dlhbc"] Nov 03 22:29:48.949011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.948919 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:48.955913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:48.955884 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-669969f6b5-ntnd8"] Nov 03 22:29:49.122113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:49.122076 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:49.122311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:49.122135 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:50.121924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:50.121889 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:50.122446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:50.121950 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:50.406739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:50.406665 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95587d49-bc69-4eba-8fce-76d67fac3d74" path="/var/lib/kubelet/pods/95587d49-bc69-4eba-8fce-76d67fac3d74/volumes" Nov 03 22:29:50.407001 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:50.406974 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c299215d-4efd-4a19-9550-ed59496f9325" path="/var/lib/kubelet/pods/c299215d-4efd-4a19-9550-ed59496f9325/volumes" Nov 03 22:29:51.122714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:51.122668 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:51.123238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:51.122733 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:51.624201 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:29:51.624156 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:29:52.122716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:52.122669 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:52.123159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:52.122739 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:53.122143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:53.122098 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:29:53.122457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:53.122162 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:29:54.125456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:54.125424 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:54.125871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:54.125688 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:29:54.126346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:29:54.126330 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:30:21.622092 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:30:21.622012 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:30:51.622234 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:30:51.622197 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:31:21.622500 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:31:21.622464 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:31:46.752928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.752846 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv"] Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753219 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c299215d-4efd-4a19-9550-ed59496f9325" containerName="authorino" Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753232 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c299215d-4efd-4a19-9550-ed59496f9325" containerName="authorino" Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753259 5051 cpu_manager.go:414] "containerMap: removing container" podUID="95587d49-bc69-4eba-8fce-76d67fac3d74" containerName="authorino" Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753264 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95587d49-bc69-4eba-8fce-76d67fac3d74" containerName="authorino" Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753318 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="95587d49-bc69-4eba-8fce-76d67fac3d74" containerName="authorino" Nov 03 22:31:46.753390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.753328 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c299215d-4efd-4a19-9550-ed59496f9325" containerName="authorino" Nov 03 22:31:46.756308 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.756292 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.812633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.812603 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h7bmr\"" Nov 03 22:31:46.856717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.856679 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvtn\" (UniqueName: \"kubernetes.io/projected/a2e1a9bd-ec01-49a9-92ef-6688291fb567-kube-api-access-mbvtn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.856717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.856729 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.856976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.856857 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.856976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.856916 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.856976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.856954 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.857159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.857135 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.862120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.862096 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 22:31:46.862120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.862111 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:31:46.862300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.862130 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:31:46.864036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.864015 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv"] Nov 03 22:31:46.958046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958009 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvtn\" (UniqueName: \"kubernetes.io/projected/a2e1a9bd-ec01-49a9-92ef-6688291fb567-kube-api-access-mbvtn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958074 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958102 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958121 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958156 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958482 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958570 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958526 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.958608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.958576 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.960748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.960719 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e1a9bd-ec01-49a9-92ef-6688291fb567-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:46.960868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:46.960766 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e1a9bd-ec01-49a9-92ef-6688291fb567-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:47.022689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:47.022607 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvtn\" (UniqueName: \"kubernetes.io/projected/a2e1a9bd-ec01-49a9-92ef-6688291fb567-kube-api-access-mbvtn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv\" (UID: \"a2e1a9bd-ec01-49a9-92ef-6688291fb567\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:47.077339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:47.077307 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:47.452285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:47.452257 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv"] Nov 03 22:31:47.454879 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:31:47.454854 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2e1a9bd_ec01_49a9_92ef_6688291fb567.slice/crio-3a58f909298189608dc7db3aa6080772c0ffbb2e26a871af2b646fa618bc564a WatchSource:0}: Error finding container 3a58f909298189608dc7db3aa6080772c0ffbb2e26a871af2b646fa618bc564a: Status 404 returned error can't find the container with id 3a58f909298189608dc7db3aa6080772c0ffbb2e26a871af2b646fa618bc564a Nov 03 22:31:48.249873 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:48.249842 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" event={"ID":"a2e1a9bd-ec01-49a9-92ef-6688291fb567","Type":"ContainerStarted","Data":"3a58f909298189608dc7db3aa6080772c0ffbb2e26a871af2b646fa618bc564a"} Nov 03 22:31:49.256826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:49.256728 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" event={"ID":"a2e1a9bd-ec01-49a9-92ef-6688291fb567","Type":"ContainerStarted","Data":"2d3f69403afd01d90ebdeff35747146a041ddc86248bfe8393a1d16f06323eb7"} Nov 03 22:31:49.374042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:49.373959 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podStartSLOduration=1.851635621 podStartE2EDuration="3.373941729s" totalImagesPullingTime="1.522306108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:49.372751801 +0000 UTC m=+583.110096026" watchObservedRunningTime="2026-11-03 22:31:49.373941729 +0000 UTC m=+583.111285953" Nov 03 22:31:50.078002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:50.077955 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:51.622386 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:31:51.622278 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:31:54.979742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:54.979708 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l"] Nov 03 22:31:54.983213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:54.983196 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.084321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.084290 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 22:31:55.086374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.086351 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l"] Nov 03 22:31:55.133288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133253 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.133472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133398 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.133472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133439 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.133563 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133521 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.133563 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133551 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.133663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.133649 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs864\" (UniqueName: \"kubernetes.io/projected/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-kube-api-access-hs864\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235095 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235136 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235186 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235235 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235302 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs864\" (UniqueName: \"kubernetes.io/projected/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-kube-api-access-hs864\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235345 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235556 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235629 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.235726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.235706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.237762 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.237735 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.238124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.238105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.296568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.296538 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs864\" (UniqueName: \"kubernetes.io/projected/ce20667b-c9f6-4f80-81a0-f63d1b87d9d3-kube-api-access-hs864\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-4sx8l\" (UID: \"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.596045 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.595930 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:31:55.976087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:55.976063 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l"] Nov 03 22:31:55.978819 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:31:55.978778 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce20667b_c9f6_4f80_81a0_f63d1b87d9d3.slice/crio-5e07cd499c84448a742059f12f0e126db608e121ac897d5aa8b1d84a3775ae1f WatchSource:0}: Error finding container 5e07cd499c84448a742059f12f0e126db608e121ac897d5aa8b1d84a3775ae1f: Status 404 returned error can't find the container with id 5e07cd499c84448a742059f12f0e126db608e121ac897d5aa8b1d84a3775ae1f Nov 03 22:31:56.294348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:56.294269 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" event={"ID":"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3","Type":"ContainerStarted","Data":"5e07cd499c84448a742059f12f0e126db608e121ac897d5aa8b1d84a3775ae1f"} Nov 03 22:31:57.078502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.078467 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:57.090908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.090884 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:57.302813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.302769 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" event={"ID":"ce20667b-c9f6-4f80-81a0-f63d1b87d9d3","Type":"ContainerStarted","Data":"8b66c5d0e6b5c58290d2625be8d8065dd0c918ca6e47472aaa3f081b925bfb42"} Nov 03 22:31:57.313927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.313905 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:31:57.420804 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.420747 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" podStartSLOduration=2.800439783 podStartE2EDuration="3.420733803s" totalImagesPullingTime="620.29402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:57.4191251 +0000 UTC m=+591.156469307" watchObservedRunningTime="2026-11-03 22:31:57.420733803 +0000 UTC m=+591.158078029" Nov 03 22:31:57.596826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:31:57.596800 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:32:02.256889 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.256857 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw"] Nov 03 22:32:02.260515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.260497 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297581 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69267d17-a659-4d2d-ae3b-2560c2968dec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297698 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297747 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297770 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297834 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.297963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.297864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zvv\" (UniqueName: \"kubernetes.io/projected/69267d17-a659-4d2d-ae3b-2560c2968dec-kube-api-access-q5zvv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.313408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.313384 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 22:32:02.319474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.319449 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw"] Nov 03 22:32:02.398477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398387 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69267d17-a659-4d2d-ae3b-2560c2968dec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.398676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398525 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.398676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398571 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.398676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398597 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.398676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398633 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.398676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.398661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zvv\" (UniqueName: \"kubernetes.io/projected/69267d17-a659-4d2d-ae3b-2560c2968dec-kube-api-access-q5zvv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.399149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.399121 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.399314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.399294 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.399557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.399410 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.401343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.401321 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69267d17-a659-4d2d-ae3b-2560c2968dec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.401434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.401392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69267d17-a659-4d2d-ae3b-2560c2968dec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.460890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.460866 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zvv\" (UniqueName: \"kubernetes.io/projected/69267d17-a659-4d2d-ae3b-2560c2968dec-kube-api-access-q5zvv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r7qkw\" (UID: \"69267d17-a659-4d2d-ae3b-2560c2968dec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.573539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.573458 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:02.834563 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.834487 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:32:02.839972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.839949 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904009 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904056 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904146 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904185 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904217 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904297 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904333 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904352 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sq6\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.904425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.904374 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:02.932476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.932440 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:32:02.962331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:02.962300 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw"] Nov 03 22:32:02.965455 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:32:02.965392 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69267d17_a659_4d2d_ae3b_2560c2968dec.slice/crio-2dfc95e458130f549c5f08cabaa6557dd18b13f6e6716adc960f0de758c870e7 WatchSource:0}: Error finding container 2dfc95e458130f549c5f08cabaa6557dd18b13f6e6716adc960f0de758c870e7: Status 404 returned error can't find the container with id 2dfc95e458130f549c5f08cabaa6557dd18b13f6e6716adc960f0de758c870e7 Nov 03 22:32:03.005302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005279 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005324 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005348 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005400 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005480 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005512 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005586 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005619 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sq6\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005671 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005745 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.005866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005824 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.006018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.005968 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.006293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.006267 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.008075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.008049 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.008195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.008179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.066748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.066716 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.066881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.066765 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sq6\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn8gd\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.154344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.154315 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:03.335521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.335491 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" event={"ID":"69267d17-a659-4d2d-ae3b-2560c2968dec","Type":"ContainerStarted","Data":"2dfc95e458130f549c5f08cabaa6557dd18b13f6e6716adc960f0de758c870e7"} Nov 03 22:32:03.534229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.534157 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:32:03.538159 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:32:03.538132 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f81a16_6fe8_4468_8f5d_4c6dbeba3fef.slice/crio-c10c51644fd1e00d1b2d579efc650218ee8a194e51b9d8dae22afa418af8b9ae WatchSource:0}: Error finding container c10c51644fd1e00d1b2d579efc650218ee8a194e51b9d8dae22afa418af8b9ae: Status 404 returned error can't find the container with id c10c51644fd1e00d1b2d579efc650218ee8a194e51b9d8dae22afa418af8b9ae Nov 03 22:32:03.541313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.541270 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:03.541391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.541366 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:03.541490 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:03.541471 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:04.343869 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.343832 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" event={"ID":"69267d17-a659-4d2d-ae3b-2560c2968dec","Type":"ContainerStarted","Data":"b230efdffe7bbad0c9727827a5e8e112486b0b9960de74a14da7ab78f5f55338"} Nov 03 22:32:04.345547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.345522 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" event={"ID":"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef","Type":"ContainerStarted","Data":"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865"} Nov 03 22:32:04.345688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.345569 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" event={"ID":"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef","Type":"ContainerStarted","Data":"c10c51644fd1e00d1b2d579efc650218ee8a194e51b9d8dae22afa418af8b9ae"} Nov 03 22:32:04.475287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.475224 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" podStartSLOduration=1.8490721940000001 podStartE2EDuration="2.475201988s" totalImagesPullingTime="626.129794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:04.468469595 +0000 UTC m=+598.205813822" watchObservedRunningTime="2026-11-03 22:32:04.475201988 +0000 UTC m=+598.212546214" Nov 03 22:32:04.580017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.579086 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:04.590680 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:04.590629 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podStartSLOduration=2.590613602 podStartE2EDuration="2.590613602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:04.588798371 +0000 UTC m=+598.326142596" watchObservedRunningTime="2026-11-03 22:32:04.590613602 +0000 UTC m=+598.327957906" Nov 03 22:32:05.155024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:05.154972 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:05.164016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:05.156884 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:32:05.164016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:05.156939 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:32:05.596463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:05.596366 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:32:05.610255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:05.610228 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:32:06.155140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:06.155103 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:32:06.155320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:06.155158 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:32:06.406814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:06.406737 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-4sx8l" Nov 03 22:32:06.516606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:06.516579 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:32:06.516891 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:06.516875 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:32:07.155502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:07.155463 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:32:07.155879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:07.155513 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:32:08.154688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:08.154653 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:32:08.154876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:08.154712 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:32:09.018456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.018419 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wvgk"] Nov 03 22:32:09.024344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.024323 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085682 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085724 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085753 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb10c14-99bd-468d-932f-a4cb23eec994-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085823 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvp6\" (UniqueName: \"kubernetes.io/projected/feb10c14-99bd-468d-932f-a4cb23eec994-kube-api-access-4tvp6\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085876 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.086014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.085903 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.089768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.089737 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wvgk"] Nov 03 22:32:09.134941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.134898 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 22:32:09.171125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.171087 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:32:09.172015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.171959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:32:09.186576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.186535 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.186845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.186821 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.186974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.186958 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb10c14-99bd-468d-932f-a4cb23eec994-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.187180 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.187161 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvp6\" (UniqueName: \"kubernetes.io/projected/feb10c14-99bd-468d-932f-a4cb23eec994-kube-api-access-4tvp6\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.187321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.187306 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.187429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.187416 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.188214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.188184 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.188558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.188526 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.189320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.189292 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.190694 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.190668 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb10c14-99bd-468d-932f-a4cb23eec994-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.191536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.191516 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb10c14-99bd-468d-932f-a4cb23eec994-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.250084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.250052 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvp6\" (UniqueName: \"kubernetes.io/projected/feb10c14-99bd-468d-932f-a4cb23eec994-kube-api-access-4tvp6\") pod \"echo-simulator-kserve-64b6f8f96c-2wvgk\" (UID: \"feb10c14-99bd-468d-932f-a4cb23eec994\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.337608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.337500 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:09.759233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:09.759209 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wvgk"] Nov 03 22:32:09.761282 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:32:09.761255 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb10c14_99bd_468d_932f_a4cb23eec994.slice/crio-a8f65a31b8dfa5cbb015324b61eab4746452d9ba8636988bc4d25d0a68e2ead3 WatchSource:0}: Error finding container a8f65a31b8dfa5cbb015324b61eab4746452d9ba8636988bc4d25d0a68e2ead3: Status 404 returned error can't find the container with id a8f65a31b8dfa5cbb015324b61eab4746452d9ba8636988bc4d25d0a68e2ead3 Nov 03 22:32:10.412540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:10.412504 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" event={"ID":"feb10c14-99bd-468d-932f-a4cb23eec994","Type":"ContainerStarted","Data":"a8f65a31b8dfa5cbb015324b61eab4746452d9ba8636988bc4d25d0a68e2ead3"} Nov 03 22:32:11.155438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:11.155394 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn8gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:11.155630 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:11.155480 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:12.140150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.140122 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:12.140632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.140505 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:12.141527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.141502 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:32:12.440397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.440359 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" event={"ID":"feb10c14-99bd-468d-932f-a4cb23eec994","Type":"ContainerStarted","Data":"ea3704ef15f87badeb1bf9c581b5ab43456616cdccb42d942e5708086a0f85be"} Nov 03 22:32:12.562324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.562268 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" podStartSLOduration=2.302890381 podStartE2EDuration="4.562254989s" totalImagesPullingTime="2.259364608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:12.559798886 +0000 UTC m=+606.297143111" watchObservedRunningTime="2026-11-03 22:32:12.562254989 +0000 UTC m=+606.299599213" Nov 03 22:32:12.574042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.574016 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:12.586576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:12.586556 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:13.338215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:13.338184 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:13.463729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:13.463692 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" Nov 03 22:32:17.952242 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:17.952206 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:32:17.955569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:17.955553 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074159 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074204 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074231 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074304 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074359 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074434 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4r5\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074541 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.074733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.074574 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.163712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.163677 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:32:18.176053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176025 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176065 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176085 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176103 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176127 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176154 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176233 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4r5\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176521 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176610 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176825 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.176847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.176832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.177103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.177084 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.178858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.178832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.178976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.178952 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.236733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.236672 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.237104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.237080 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4r5\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fsjfb\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.268351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.268325 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:18.747724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.747696 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:32:18.749907 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:32:18.749875 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e7683c_3605_4397_8157_074947210385.slice/crio-b3c2a7f4310ff3409ce19ea52f5240e8b4cf691607058f8afb6a325cf1975c2b WatchSource:0}: Error finding container b3c2a7f4310ff3409ce19ea52f5240e8b4cf691607058f8afb6a325cf1975c2b: Status 404 returned error can't find the container with id b3c2a7f4310ff3409ce19ea52f5240e8b4cf691607058f8afb6a325cf1975c2b Nov 03 22:32:18.752269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.752233 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:18.752358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.752306 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:18.752358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:18.752335 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:19.338297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.338269 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:19.350930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.350904 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:19.487825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.487786 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" event={"ID":"d1e7683c-3605-4397-8157-074947210385","Type":"ContainerStarted","Data":"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304"} Nov 03 22:32:19.488045 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.487844 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" event={"ID":"d1e7683c-3605-4397-8157-074947210385","Type":"ContainerStarted","Data":"b3c2a7f4310ff3409ce19ea52f5240e8b4cf691607058f8afb6a325cf1975c2b"} Nov 03 22:32:19.499631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.499592 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" Nov 03 22:32:19.609949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:19.609894 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podStartSLOduration=2.60987991 podStartE2EDuration="2.60987991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:19.608451167 +0000 UTC m=+613.345795393" watchObservedRunningTime="2026-11-03 22:32:19.60987991 +0000 UTC m=+613.347224135" Nov 03 22:32:20.275704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:20.275671 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:20.276085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:20.276057 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:20.276262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:20.276235 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:21.270014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:21.269363 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:21.270014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:21.269421 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:21.624688 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:32:21.624650 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:32:22.271437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:22.271396 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:22.272130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:22.272085 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:23.269529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:23.269493 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:23.269847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:23.269803 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:24.279023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:24.273388 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:24.279023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:24.273461 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:25.269216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:25.269179 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 22:32:25.269476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:25.269232 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 22:32:27.269350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:27.269303 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:27.269835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:27.269390 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:28.311157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:28.311124 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fsjfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:28.311684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:28.311180 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:29.064604 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:29.064572 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:29.064893 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:29.064870 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:29.065626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:29.065608 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:32:33.063562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.063524 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:32:33.067374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.067355 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.146835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.146792 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.146864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjp7\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.146957 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147019 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147102 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147155 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147259 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.147339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.147317 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.170633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.170594 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:32:33.248333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248338 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248377 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248445 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248679 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjp7\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248780 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248866 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.248930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248918 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.249079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.248953 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.249148 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.249109 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.249204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.249170 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.251106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.251073 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.251364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.251347 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.309183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.309152 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.309424 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.309403 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjp7\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9gvgr\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.381328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.381298 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:33.812888 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.812858 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:32:33.815078 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:32:33.815051 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068095c9_59cb_4e19_8183_784135e57e88.slice/crio-f2eaf0b693280ad8f1f642128f09948b01e0a048b5894974d5705c1efa060136 WatchSource:0}: Error finding container f2eaf0b693280ad8f1f642128f09948b01e0a048b5894974d5705c1efa060136: Status 404 returned error can't find the container with id f2eaf0b693280ad8f1f642128f09948b01e0a048b5894974d5705c1efa060136 Nov 03 22:32:33.817191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.817161 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:33.817257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.817228 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:33.817292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:33.817257 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:34.595373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:34.595337 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" event={"ID":"068095c9-59cb-4e19-8183-784135e57e88","Type":"ContainerStarted","Data":"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d"} Nov 03 22:32:34.595858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:34.595388 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" event={"ID":"068095c9-59cb-4e19-8183-784135e57e88","Type":"ContainerStarted","Data":"f2eaf0b693280ad8f1f642128f09948b01e0a048b5894974d5705c1efa060136"} Nov 03 22:32:35.382217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:35.382171 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:35.383879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:35.383849 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:35.384041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:35.383906 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:36.382601 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:36.382562 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:36.383047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:36.382632 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:37.382603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:37.382558 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:37.382780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:37.382633 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:38.381816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:38.381783 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:38.382035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:38.381850 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:39.382840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:39.382801 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:39.383283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:39.382882 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:40.382155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:40.382120 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9gvgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:32:40.382341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:40.382176 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 22:32:42.089194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:42.089148 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:42.089709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:42.089691 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:42.090425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:42.090408 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:32:42.261394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:32:42.261337 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podStartSLOduration=9.261323291 podStartE2EDuration="9.261323291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:34.781348834 +0000 UTC m=+628.518693060" watchObservedRunningTime="2026-11-03 22:32:42.261323291 +0000 UTC m=+635.998667591" Nov 03 22:32:51.627240 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:32:51.627204 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:33:21.622929 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:33:21.622846 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:33:43.935706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:43.935674 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:33:43.939445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:43.939429 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:43.995972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:43.995944 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b64e2e-openshift-default-dockercfg-g4cxs\"" Nov 03 22:33:44.044197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044158 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044209 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044334 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmzm\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044478 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044529 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044576 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044593 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.044663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.044649 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.045345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.045321 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:33:44.146244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146213 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146267 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146293 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146495 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmzm\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146587 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146659 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146696 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.146878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.146857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.147039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.147013 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.147152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.147047 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.147346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.147327 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.149073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.149056 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.149164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.149114 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.208363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.208281 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmzm\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.208523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.208400 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.250423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.250391 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 22:33:44.253116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.253098 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:44.255123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.255109 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.320916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.320867 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7afb1c-openshift-default-dockercfg-nlmnh\"" Nov 03 22:33:44.322506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.322470 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349358 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349445 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349480 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349515 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349574 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349604 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcv6\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.350653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.349637 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460603 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460714 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460744 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460778 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460810 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.460883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460863 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcv6\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.461528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460896 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.461528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.460950 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.462508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.462480 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.462776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.462757 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.463069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.463049 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.464022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.463564 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.464943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.464442 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.479329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.476590 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.479777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.479726 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.525032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.525007 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.525176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.525123 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcv6\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.579267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.579234 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:44.725228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.725035 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:33:44.730218 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:33:44.730181 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a5d2fb_5ff7_43f9_a1a5_76de45fc1b92.slice/crio-81c18db6b99f3b04a107e20e2a31aa3e9b3a848e981cf574573ce3fb7ce52073 WatchSource:0}: Error finding container 81c18db6b99f3b04a107e20e2a31aa3e9b3a848e981cf574573ce3fb7ce52073: Status 404 returned error can't find the container with id 81c18db6b99f3b04a107e20e2a31aa3e9b3a848e981cf574573ce3fb7ce52073 Nov 03 22:33:44.732488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.732452 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:44.732593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.732532 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:44.732593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.732564 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:44.952779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.952750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerStarted","Data":"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325"} Nov 03 22:33:44.953185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.952811 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerStarted","Data":"81c18db6b99f3b04a107e20e2a31aa3e9b3a848e981cf574573ce3fb7ce52073"} Nov 03 22:33:44.996038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:44.990228 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 22:33:45.004155 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:33:45.004124 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb280a08d_b941_4f52_ad36_cac35ee9afb8.slice/crio-d24fed7df2a94331d39d5dc7c34602af07ba683bc7a8be812f6578d34e20695b WatchSource:0}: Error finding container d24fed7df2a94331d39d5dc7c34602af07ba683bc7a8be812f6578d34e20695b: Status 404 returned error can't find the container with id d24fed7df2a94331d39d5dc7c34602af07ba683bc7a8be812f6578d34e20695b Nov 03 22:33:45.011890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.011849 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.012055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.011931 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.012055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.011970 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.077319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.077199 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podStartSLOduration=2.077180744 podStartE2EDuration="2.077180744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:45.074511074 +0000 UTC m=+698.811855298" watchObservedRunningTime="2026-11-03 22:33:45.077180744 +0000 UTC m=+698.814524969" Nov 03 22:33:45.254309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.254229 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:45.258310 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.258284 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 22:33:45.261850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.261829 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:45.262267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.262013 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.314510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.314481 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-cbd26c-openshift-default-dockercfg-5rskr\"" Nov 03 22:33:45.326827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.326797 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 22:33:45.370397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370365 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370440 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370471 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370495 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370626 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370738 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370810 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.370872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.370857 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.471809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471776 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.471809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471816 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471845 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471869 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471934 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.471971 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472125 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472197 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472241 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472255 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472324 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472378 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472601 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.472655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.472608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.474361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.474344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.474482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.474467 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.533332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.533260 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.535039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.535020 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.579549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.579517 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:45.959185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.959150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerStarted","Data":"952b66e07e1c47c9784d3938e7f65f090ab2820e5b27b587ad3b9a48ec9b070e"} Nov 03 22:33:45.959665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.959205 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerStarted","Data":"d24fed7df2a94331d39d5dc7c34602af07ba683bc7a8be812f6578d34e20695b"} Nov 03 22:33:45.959665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.959307 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:45.960288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:45.960270 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:33:46.010934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.010881 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 22:33:46.018627 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:33:46.018574 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487d8d62_e8c3_4197_a94c_75de196ef9e0.slice/crio-e0a8a2b1d778cd881103361b9644f94b04f5a589b377c1ce784a8000c19b9557 WatchSource:0}: Error finding container e0a8a2b1d778cd881103361b9644f94b04f5a589b377c1ce784a8000c19b9557: Status 404 returned error can't find the container with id e0a8a2b1d778cd881103361b9644f94b04f5a589b377c1ce784a8000c19b9557 Nov 03 22:33:46.021092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.021057 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:46.021193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.021124 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:46.021193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.021153 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:46.215166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.215058 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podStartSLOduration=2.215040934 podStartE2EDuration="2.215040934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:46.090620921 +0000 UTC m=+699.827965145" watchObservedRunningTime="2026-11-03 22:33:46.215040934 +0000 UTC m=+699.952385161" Nov 03 22:33:46.580368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.580284 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:46.586105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.586074 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:46.981927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.981891 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerStarted","Data":"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144"} Nov 03 22:33:46.982426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.981954 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerStarted","Data":"e0a8a2b1d778cd881103361b9644f94b04f5a589b377c1ce784a8000c19b9557"} Nov 03 22:33:46.982426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.982095 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:46.983076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:46.983054 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:33:47.108241 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:47.108195 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podStartSLOduration=2.108181775 podStartE2EDuration="2.108181775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:47.107150736 +0000 UTC m=+700.844494973" watchObservedRunningTime="2026-11-03 22:33:47.108181775 +0000 UTC m=+700.845526057" Nov 03 22:33:47.580064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:47.580032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:47.586113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:47.586089 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:47.986549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:47.986517 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:47.987422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:33:47.987407 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:33:51.622161 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:33:51.622125 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:34:00.664334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.664300 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 22:34:00.668097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.668077 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.728521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.728487 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhj5\" (UniqueName: \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.728712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.728666 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.788388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.788352 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7afb1c\"" Nov 03 22:34:00.788760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.788741 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7afb1c-dockercfg-8ndm6\"" Nov 03 22:34:00.804090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.804052 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 22:34:00.829481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.829441 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhj5\" (UniqueName: \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.829692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.829531 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.830185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.830166 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.830442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.830422 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 22:34:00.834124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.834111 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:00.871606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.871581 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 22:34:00.890466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.890439 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhj5\" (UniqueName: \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\") pod \"payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:00.930437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.930372 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:00.930547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.930532 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6mc\" (UniqueName: \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:00.981628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:00.981587 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:01.034700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.034664 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:01.034848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.034830 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6mc\" (UniqueName: \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:01.035666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.035639 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:01.095916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.095890 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6mc\" (UniqueName: \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\") pod \"payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:01.147154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.147123 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:01.362106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.362080 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 22:34:01.363893 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:01.363871 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ef8fe2_a9eb_4d08_9c02_eb2150f04ebc.slice/crio-79c0c806af3daeec272435d5776494d256660f8c367ae5974c7212fdde5c038a WatchSource:0}: Error finding container 79c0c806af3daeec272435d5776494d256660f8c367ae5974c7212fdde5c038a: Status 404 returned error can't find the container with id 79c0c806af3daeec272435d5776494d256660f8c367ae5974c7212fdde5c038a Nov 03 22:34:01.366278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.366263 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:34:01.523974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:01.523907 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 22:34:01.526797 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:01.526770 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59f895d_e385_45ea_830d_289b0d565c31.slice/crio-cae966b9d514ce39613a16c27100039471d888c7de58470e6f913dd912b43c38 WatchSource:0}: Error finding container cae966b9d514ce39613a16c27100039471d888c7de58470e6f913dd912b43c38: Status 404 returned error can't find the container with id cae966b9d514ce39613a16c27100039471d888c7de58470e6f913dd912b43c38 Nov 03 22:34:02.068572 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:02.068533 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" event={"ID":"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc","Type":"ContainerStarted","Data":"79c0c806af3daeec272435d5776494d256660f8c367ae5974c7212fdde5c038a"} Nov 03 22:34:02.069814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:02.069794 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" event={"ID":"a59f895d-e385-45ea-830d-289b0d565c31","Type":"ContainerStarted","Data":"cae966b9d514ce39613a16c27100039471d888c7de58470e6f913dd912b43c38"} Nov 03 22:34:05.621979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.621954 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 22:34:05.626847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.626824 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.680457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.680367 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7540da\"" Nov 03 22:34:05.706458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.706431 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.706621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.706509 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c7d\" (UniqueName: \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.727030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.726950 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7540da-dockercfg-jztv4\"" Nov 03 22:34:05.729698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.729676 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 22:34:05.771564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.771536 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 22:34:05.775349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.775329 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:05.808324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.808288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.808501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.808412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c7d\" (UniqueName: \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.809287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.809256 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.846193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.846157 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 22:34:05.879849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.879811 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c7d\" (UniqueName: \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\") pod \"payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:05.910046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.909972 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:05.910228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.910187 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tpk\" (UniqueName: \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:05.953928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:05.953886 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:06.012258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.012163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:06.012421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.012330 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tpk\" (UniqueName: \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:06.014255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.013469 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:06.075264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.075234 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tpk\" (UniqueName: \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\") pod \"payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:06.090116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.090081 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:06.109838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.109807 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" event={"ID":"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc","Type":"ContainerStarted","Data":"4b2db9da3d2d26840688a294a0a1046ad6a8c8dde0962f0e952b7904556a4a32"} Nov 03 22:34:06.109973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.109876 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:06.111691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.111663 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" event={"ID":"a59f895d-e385-45ea-830d-289b0d565c31","Type":"ContainerStarted","Data":"8aee8e2c60b852d72d60e744ff02d5919c385437c0bfbd3bda0c887ff2ea5092"} Nov 03 22:34:06.111792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.111735 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:06.225894 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.225845 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" podStartSLOduration=1.990853462 podStartE2EDuration="6.225830052s" totalImagesPullingTime="4.23497659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:06.22443921 +0000 UTC m=+719.961783434" watchObservedRunningTime="2026-11-03 22:34:06.225830052 +0000 UTC m=+719.963174276" Nov 03 22:34:06.348196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.345915 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 22:34:06.351251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.351189 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" podStartSLOduration=2.414179184 podStartE2EDuration="6.351169615s" totalImagesPullingTime="3.936990431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:06.341745891 +0000 UTC m=+720.079090117" watchObservedRunningTime="2026-11-03 22:34:06.351169615 +0000 UTC m=+720.088513840" Nov 03 22:34:06.356724 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:06.356696 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84397155_43d0_4d44_b3f2_285f6c39ba81.slice/crio-16db0057280f00ef6c026c6ce00d953993c74daf686f0ecb098ccb82bca52e45 WatchSource:0}: Error finding container 16db0057280f00ef6c026c6ce00d953993c74daf686f0ecb098ccb82bca52e45: Status 404 returned error can't find the container with id 16db0057280f00ef6c026c6ce00d953993c74daf686f0ecb098ccb82bca52e45 Nov 03 22:34:06.475787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:06.475686 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 22:34:06.480476 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:06.480447 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode541be41_d501_4287_8a8d_6d5695aaf730.slice/crio-edb80754a69888fe7f801ce6544b3dc442db4bd203192ea7038556fdf1503267 WatchSource:0}: Error finding container edb80754a69888fe7f801ce6544b3dc442db4bd203192ea7038556fdf1503267: Status 404 returned error can't find the container with id edb80754a69888fe7f801ce6544b3dc442db4bd203192ea7038556fdf1503267 Nov 03 22:34:07.120152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.120108 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" event={"ID":"e541be41-d501-4287-8a8d-6d5695aaf730","Type":"ContainerStarted","Data":"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20"} Nov 03 22:34:07.120645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.120178 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" event={"ID":"e541be41-d501-4287-8a8d-6d5695aaf730","Type":"ContainerStarted","Data":"edb80754a69888fe7f801ce6544b3dc442db4bd203192ea7038556fdf1503267"} Nov 03 22:34:07.120645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.120304 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:07.121909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.121879 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" event={"ID":"84397155-43d0-4d44-b3f2-285f6c39ba81","Type":"ContainerStarted","Data":"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6"} Nov 03 22:34:07.122044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.121926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" event={"ID":"84397155-43d0-4d44-b3f2-285f6c39ba81","Type":"ContainerStarted","Data":"16db0057280f00ef6c026c6ce00d953993c74daf686f0ecb098ccb82bca52e45"} Nov 03 22:34:07.122136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.122120 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:07.237376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.237326 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" podStartSLOduration=2.237309286 podStartE2EDuration="2.237309286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:07.235459667 +0000 UTC m=+720.972803904" watchObservedRunningTime="2026-11-03 22:34:07.237309286 +0000 UTC m=+720.974653511" Nov 03 22:34:07.351459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:07.351407 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" podStartSLOduration=2.351394503 podStartE2EDuration="2.351394503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:07.350286319 +0000 UTC m=+721.087630547" watchObservedRunningTime="2026-11-03 22:34:07.351394503 +0000 UTC m=+721.088738727" Nov 03 22:34:08.172813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.172781 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 22:34:08.176613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.176594 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.235871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.235841 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbt9x\" (UniqueName: \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.236029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.235913 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.291011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.287063 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-1d31f8-dockercfg-xrthd\"" Nov 03 22:34:08.291011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.287168 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1d31f8\"" Nov 03 22:34:08.313751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.311077 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 22:34:08.325306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.325273 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 22:34:08.329080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.329066 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.336941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.336915 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbt9x\" (UniqueName: \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.337076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.337011 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.337683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.337664 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.388474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.388445 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 22:34:08.397863 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.397830 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbt9x\" (UniqueName: \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\") pod \"payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.438042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.437951 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kjq\" (UniqueName: \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.438170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.438112 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.489270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.489241 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:08.538759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.538721 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kjq\" (UniqueName: \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.538930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.538912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.539693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.539675 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.618504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.618473 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kjq\" (UniqueName: \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\") pod \"payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.642464 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.642438 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:08.880434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:08.880371 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 22:34:08.882843 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:08.882816 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfadf4df_4b62_46c3_a3ce_46e4785e8f1e.slice/crio-11966452b34679fddfee4bc514bf7b7e0a7c52d06f1752db59d76950f5f40105 WatchSource:0}: Error finding container 11966452b34679fddfee4bc514bf7b7e0a7c52d06f1752db59d76950f5f40105: Status 404 returned error can't find the container with id 11966452b34679fddfee4bc514bf7b7e0a7c52d06f1752db59d76950f5f40105 Nov 03 22:34:09.029374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.029353 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 22:34:09.031634 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:09.031606 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef12284d_6252_4e47_9823_4ed240d9c58e.slice/crio-b5123360311ce5f8863221f086ccde787aaea63d474e272b1142678a5be45386 WatchSource:0}: Error finding container b5123360311ce5f8863221f086ccde787aaea63d474e272b1142678a5be45386: Status 404 returned error can't find the container with id b5123360311ce5f8863221f086ccde787aaea63d474e272b1142678a5be45386 Nov 03 22:34:09.134113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.134084 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" event={"ID":"ef12284d-6252-4e47-9823-4ed240d9c58e","Type":"ContainerStarted","Data":"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52"} Nov 03 22:34:09.134248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.134143 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" event={"ID":"ef12284d-6252-4e47-9823-4ed240d9c58e","Type":"ContainerStarted","Data":"b5123360311ce5f8863221f086ccde787aaea63d474e272b1142678a5be45386"} Nov 03 22:34:09.134248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.134188 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:09.135776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.135750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" event={"ID":"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e","Type":"ContainerStarted","Data":"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a"} Nov 03 22:34:09.135866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.135795 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" event={"ID":"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e","Type":"ContainerStarted","Data":"11966452b34679fddfee4bc514bf7b7e0a7c52d06f1752db59d76950f5f40105"} Nov 03 22:34:09.135866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.135818 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:09.250969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.250919 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" podStartSLOduration=1.25090692 podStartE2EDuration="1.25090692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:09.248323174 +0000 UTC m=+722.985667398" watchObservedRunningTime="2026-11-03 22:34:09.25090692 +0000 UTC m=+722.988251145" Nov 03 22:34:09.364157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:09.364102 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" podStartSLOduration=1.36408602 podStartE2EDuration="1.36408602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:09.363044814 +0000 UTC m=+723.100389052" watchObservedRunningTime="2026-11-03 22:34:09.36408602 +0000 UTC m=+723.101430246" Nov 03 22:34:15.582014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.581417 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 22:34:15.586621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.586595 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.628365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.628337 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 22:34:15.655190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.655162 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4ck\" (UniqueName: \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.655325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.655292 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.682013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.681281 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-164c36\"" Nov 03 22:34:15.682013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.681445 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-164c36-dockercfg-nvfrw\"" Nov 03 22:34:15.735016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.734520 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 22:34:15.739017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.738980 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.781714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.781671 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4ck\" (UniqueName: \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.781878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.781725 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.781878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.781761 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtm8\" (UniqueName: \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.781878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.781840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.782689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.782658 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.784544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.784521 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 22:34:15.843720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.843656 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4ck\" (UniqueName: \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\") pod \"payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.883547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.883508 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.883755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.883562 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtm8\" (UniqueName: \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.884599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.884573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:15.900291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.900266 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:15.945299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:15.945269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtm8\" (UniqueName: \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\") pod \"payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:16.052676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:16.052639 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:16.300519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:16.300493 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 22:34:16.303320 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:16.303293 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19dcbce_f33e_453d_a37a_d1460bafa5aa.slice/crio-0dee884413a6695ef3d4b57746a771fc98ee4a819d7498e416049830351ba9ba WatchSource:0}: Error finding container 0dee884413a6695ef3d4b57746a771fc98ee4a819d7498e416049830351ba9ba: Status 404 returned error can't find the container with id 0dee884413a6695ef3d4b57746a771fc98ee4a819d7498e416049830351ba9ba Nov 03 22:34:16.482315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:16.481500 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 22:34:16.484373 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:16.484342 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909d3a0e_b2e9_4696_b6f9_1596e320f82b.slice/crio-47392480b6eae24c561496c4590f34aa2db7aeb697653edcbb1a0b4770ae7f42 WatchSource:0}: Error finding container 47392480b6eae24c561496c4590f34aa2db7aeb697653edcbb1a0b4770ae7f42: Status 404 returned error can't find the container with id 47392480b6eae24c561496c4590f34aa2db7aeb697653edcbb1a0b4770ae7f42 Nov 03 22:34:17.124767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.124740 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 22:34:17.125197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.125072 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 22:34:17.221049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.221014 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" event={"ID":"e19dcbce-f33e-453d-a37a-d1460bafa5aa","Type":"ContainerStarted","Data":"a5ce2faef493b82c658f838798e7d97f4f92f946cf16db807107d97032c0a01e"} Nov 03 22:34:17.221225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.221072 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" event={"ID":"e19dcbce-f33e-453d-a37a-d1460bafa5aa","Type":"ContainerStarted","Data":"0dee884413a6695ef3d4b57746a771fc98ee4a819d7498e416049830351ba9ba"} Nov 03 22:34:17.221225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.221104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:17.222449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.222426 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" event={"ID":"909d3a0e-b2e9-4696-b6f9-1596e320f82b","Type":"ContainerStarted","Data":"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2"} Nov 03 22:34:17.222554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.222463 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" event={"ID":"909d3a0e-b2e9-4696-b6f9-1596e320f82b","Type":"ContainerStarted","Data":"47392480b6eae24c561496c4590f34aa2db7aeb697653edcbb1a0b4770ae7f42"} Nov 03 22:34:17.222554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.222494 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:17.485070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.473349 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" podStartSLOduration=2.473330856 podStartE2EDuration="2.473330856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:17.470561338 +0000 UTC m=+731.207905563" watchObservedRunningTime="2026-11-03 22:34:17.473330856 +0000 UTC m=+731.210675082" Nov 03 22:34:17.590355 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:17.589977 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" podStartSLOduration=2.589962244 podStartE2EDuration="2.589962244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:17.584512494 +0000 UTC m=+731.321856721" watchObservedRunningTime="2026-11-03 22:34:17.589962244 +0000 UTC m=+731.327306469" Nov 03 22:34:18.130583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:18.130556 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 22:34:18.131017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:18.130726 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 22:34:20.144477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:20.144451 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 22:34:20.144845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:20.144515 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 22:34:21.622629 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:34:21.622593 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:34:22.254048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.254011 5051 generic.go:356] "Generic (PLEG): container finished" podID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerID="952b66e07e1c47c9784d3938e7f65f090ab2820e5b27b587ad3b9a48ec9b070e" exitCode=0 Nov 03 22:34:22.254048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.254025 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerDied","Data":"952b66e07e1c47c9784d3938e7f65f090ab2820e5b27b587ad3b9a48ec9b070e"} Nov 03 22:34:22.254398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.254385 5051 scope.go:122] "RemoveContainer" containerID="952b66e07e1c47c9784d3938e7f65f090ab2820e5b27b587ad3b9a48ec9b070e" Nov 03 22:34:22.255650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.255619 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:22.255745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.255685 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:22.255745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:22.255716 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:23.320115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:23.317267 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerStarted","Data":"d14f4e3653d4fa0806fbb6e7efdbc78ea58f108e38b17d46cbeb95729b389249"} Nov 03 22:34:23.581076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:23.580405 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:34:23.581258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:23.581137 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:34:23.581258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:23.581206 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:34:24.580482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:24.580440 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:34:24.581719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:24.581120 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:34:25.581048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:25.580477 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:34:25.581048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:25.580540 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:34:26.581460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:26.581419 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:34:26.581870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:26.581495 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:34:27.582014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:27.581492 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:34:27.582014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:27.581554 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:34:28.231270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:28.231239 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 22:34:28.231455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:28.231333 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 22:34:28.585352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:28.585272 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:34:28.585835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:28.585601 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:34:28.586336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:28.586313 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 22:34:30.418113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.418081 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 22:34:30.423724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.423705 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.485882 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.485838 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-164c36-openshift-default-dockercfg-tlv7g\"" Nov 03 22:34:30.537898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.537869 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 22:34:30.602633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.602599 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.602834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.602722 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.602834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.602756 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57wm\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.602955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.602840 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.602955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.602908 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.603104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.603010 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.603104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.603056 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.603104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.603083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.603211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.603168 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704224 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704122 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704224 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704204 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704231 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704275 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704313 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704392 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704425 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57wm\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704487 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704559 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704890 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.704972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.704931 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.705220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.705201 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.705324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.705306 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.705685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.705666 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.707805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.707778 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.709547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.709522 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.772539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.772468 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:30.812022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:30.776753 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57wm\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:31.036362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:31.036268 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:31.526256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:31.525829 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 22:34:31.530477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:31.530431 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.530564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:31.530548 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.530602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:31.530585 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:32.417916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:32.417874 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" event={"ID":"b02868e4-1e87-401e-8640-71ab33a32010","Type":"ContainerStarted","Data":"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc"} Nov 03 22:34:32.418105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:32.417933 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" event={"ID":"b02868e4-1e87-401e-8640-71ab33a32010","Type":"ContainerStarted","Data":"bf656dca803b3596270237c93518a24b61f44b2419628ebb09c687382715fd68"} Nov 03 22:34:32.641200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:32.641152 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podStartSLOduration=2.641139292 podStartE2EDuration="2.641139292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:32.589280642 +0000 UTC m=+746.326624866" watchObservedRunningTime="2026-11-03 22:34:32.641139292 +0000 UTC m=+746.378483510" Nov 03 22:34:33.037240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:33.037200 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:33.038771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:33.038749 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:34:33.038829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:33.038797 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 22:34:34.037515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:34.037480 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:34:34.037913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:34.037537 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 22:34:35.037111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:35.037074 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:34:35.037307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:35.037134 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 22:34:36.037015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:36.036952 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:34:36.037403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:36.037048 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 22:34:37.037033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:37.036975 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:34:37.037420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:37.037055 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 22:34:38.041555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:38.041521 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:38.041955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:38.041918 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:38.042719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:38.042696 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 22:34:39.462123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.462085 5051 generic.go:356] "Generic (PLEG): container finished" podID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerID="eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325" exitCode=0 Nov 03 22:34:39.462648 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.462158 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerDied","Data":"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325"} Nov 03 22:34:39.462830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.462814 5051 scope.go:122] "RemoveContainer" containerID="eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325" Nov 03 22:34:39.464010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.463968 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:39.464079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.464047 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:39.464307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:39.464084 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:40.489193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:40.489163 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerStarted","Data":"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c"} Nov 03 22:34:41.253832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:41.253801 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:34:41.255086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:41.255065 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:34:41.255140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:41.255118 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 22:34:42.253942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:42.253909 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:34:42.254363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:42.253977 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 22:34:43.254278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:43.254242 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:34:43.254660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:43.254305 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 22:34:44.120450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.120418 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 22:34:44.124439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.124423 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.181478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.181428 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"default-dockercfg-86zlr\"" Nov 03 22:34:44.227827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.227793 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.227976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.227841 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.227976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.227965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.228086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.228064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgpz\" (UniqueName: \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.228123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.228104 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.228155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.228132 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.230899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.230876 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"openshift-service-ca.crt\"" Nov 03 22:34:44.230899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.230885 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"kube-root-ca.crt\"" Nov 03 22:34:44.233672 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.233655 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 22:34:44.254317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.254294 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:34:44.254614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.254346 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 22:34:44.286079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.281350 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"simulator-w4-32449d-kserve-self-signed-certs\"" Nov 03 22:34:44.329345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329319 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329356 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329409 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329534 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329577 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgpz\" (UniqueName: \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329755 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.329815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.329775 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.330317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.330294 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.332237 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.332212 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.332433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.332411 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.395349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.395270 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgpz\" (UniqueName: \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\") pod \"simulator-w4-32449d-kserve-574ccbf495-ftt4s\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.436691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.436666 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:44.848842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:44.848815 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 22:34:44.851105 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:34:44.851074 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f4719b5_0280_4bab_8781_bab28aec85f3.slice/crio-77254700da84d067952c444fdfe7beb0649416c248de6dcd95ea2697b47a651a WatchSource:0}: Error finding container 77254700da84d067952c444fdfe7beb0649416c248de6dcd95ea2697b47a651a: Status 404 returned error can't find the container with id 77254700da84d067952c444fdfe7beb0649416c248de6dcd95ea2697b47a651a Nov 03 22:34:45.253449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.253419 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:34:45.253628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.253472 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 22:34:45.524369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.524277 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" event={"ID":"7f4719b5-0280-4bab-8781-bab28aec85f3","Type":"ContainerStarted","Data":"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b"} Nov 03 22:34:45.524369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.524328 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" event={"ID":"7f4719b5-0280-4bab-8781-bab28aec85f3","Type":"ContainerStarted","Data":"77254700da84d067952c444fdfe7beb0649416c248de6dcd95ea2697b47a651a"} Nov 03 22:34:45.640970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.640908 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podStartSLOduration=1.640890527 podStartE2EDuration="1.640890527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:45.640138585 +0000 UTC m=+759.377482811" watchObservedRunningTime="2026-11-03 22:34:45.640890527 +0000 UTC m=+759.378234752" Nov 03 22:34:45.959967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:45.959938 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:34:46.258249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:46.258178 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:34:46.437401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:46.437373 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:46.531408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:46.531321 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:34:51.622452 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:34:51.622405 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:34:54.437659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:54.437625 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:54.451270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:54.451247 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:54.615250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:54.615221 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:34:56.618779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.618748 5051 generic.go:356] "Generic (PLEG): container finished" podID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerID="3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144" exitCode=0 Nov 03 22:34:56.619335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.618827 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerDied","Data":"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144"} Nov 03 22:34:56.619335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.619304 5051 scope.go:122] "RemoveContainer" containerID="3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144" Nov 03 22:34:56.620674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.620648 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:56.620758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.620716 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:56.620758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:56.620746 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:57.629409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:57.629366 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerStarted","Data":"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23"} Nov 03 22:34:58.586217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:58.586169 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:34:58.586418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:58.586293 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:34:58.586418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:58.586335 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:34:59.581401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:59.581364 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:34:59.581857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:34:59.581417 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:35:00.581365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.581333 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:35:00.581560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.581398 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:35:00.679011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.678598 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:35:00.683416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.683389 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728074 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728107 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728123 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728247 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728283 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgs4c\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728380 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728408 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728440 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.728526 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.728511 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.783014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.776960 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:35:00.829687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829651 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgs4c\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829743 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829772 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829814 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829849 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829933 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.829967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.829964 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830147 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830236 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830237 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830315 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830368 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830387 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.830629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.830604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.832436 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.832380 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.832699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.832680 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.923699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.923667 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgs4c\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.925088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.925057 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:00.997360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:00.997300 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:01.431489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.431465 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:35:01.433655 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:35:01.433627 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82178681_caca_4f12_a0ad_a657a9277729.slice/crio-5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473 WatchSource:0}: Error finding container 5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473: Status 404 returned error can't find the container with id 5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473 Nov 03 22:35:01.436167 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.436133 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:35:01.436260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.436205 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:35:01.436260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.436233 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:35:01.581016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.580331 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:35:01.581016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.580389 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:35:01.654611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.654571 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" event={"ID":"82178681-caca-4f12-a0ad-a657a9277729","Type":"ContainerStarted","Data":"dbf4beb5af35aea8da9ad9e4a8e1b9c490cd78b9a97a568d26817baaf3118d6f"} Nov 03 22:35:01.655032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.654635 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" event={"ID":"82178681-caca-4f12-a0ad-a657a9277729","Type":"ContainerStarted","Data":"5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473"} Nov 03 22:35:01.998176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:01.998086 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:02.640696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.640664 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:35:02.641137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.641117 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:35:02.642389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.642368 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 22:35:02.804122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.804074 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podStartSLOduration=2.804060126 podStartE2EDuration="2.804060126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:01.778290955 +0000 UTC m=+775.515635191" watchObservedRunningTime="2026-11-03 22:35:02.804060126 +0000 UTC m=+776.541404360" Nov 03 22:35:02.999512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.999427 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:35:02.999512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:02.999494 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:35:03.998408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:03.998358 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:35:03.998919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:03.998427 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:35:04.998181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:04.998142 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:35:04.998376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:04.998203 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:35:05.998170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:05.998132 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:35:05.998574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:05.998197 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:35:06.998354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:06.998318 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:35:06.998725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:06.998393 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:35:07.122617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.122582 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 22:35:07.127680 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.127658 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.181666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.181639 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"kube-root-ca.crt\"" Nov 03 22:35:07.182204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.182186 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"openshift-service-ca.crt\"" Nov 03 22:35:07.227002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.226958 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.227154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.227005 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.227154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.227128 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.227252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.227239 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.227287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.227262 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s85g\" (UniqueName: \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.227342 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.227329 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.231128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.231097 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"default-dockercfg-6f5cb\"" Nov 03 22:35:07.231266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.231238 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"simulator-w1-7afb1c-kserve-self-signed-certs\"" Nov 03 22:35:07.235610 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.235584 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 22:35:07.328910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.328818 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.329643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.329619 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.330246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.330225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.331435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.331411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.331611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.331592 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s85g\" (UniqueName: \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.331775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.331760 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.332283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.332264 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.332684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.332664 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.333089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.333069 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.354429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.354402 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.368043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.368015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.397033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.396982 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s85g\" (UniqueName: \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\") pod \"simulator-w1-7afb1c-kserve-57b5f79484-2g2xk\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.440097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.440050 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:07.837396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:07.837374 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 22:35:07.839023 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:35:07.838976 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3fd7064_cd70_4b2e_a9bf_71d4c4d97499.slice/crio-4ae2e55c96d17700346f11f47c843c5ed6a930878c8ca6e12e404f15824ef0b5 WatchSource:0}: Error finding container 4ae2e55c96d17700346f11f47c843c5ed6a930878c8ca6e12e404f15824ef0b5: Status 404 returned error can't find the container with id 4ae2e55c96d17700346f11f47c843c5ed6a930878c8ca6e12e404f15824ef0b5 Nov 03 22:35:08.002451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.002426 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:08.002886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.002794 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:08.003343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.003327 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:35:08.724168 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.724129 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" event={"ID":"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499","Type":"ContainerStarted","Data":"c32dc26048f845f28e3ab1293c353b5d168285ee9aefb47d96477d2630c96118"} Nov 03 22:35:08.724168 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.724168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" event={"ID":"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499","Type":"ContainerStarted","Data":"4ae2e55c96d17700346f11f47c843c5ed6a930878c8ca6e12e404f15824ef0b5"} Nov 03 22:35:08.842161 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:08.842114 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podStartSLOduration=1.842101175 podStartE2EDuration="1.842101175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:08.840141642 +0000 UTC m=+782.577485866" watchObservedRunningTime="2026-11-03 22:35:08.842101175 +0000 UTC m=+782.579445398" Nov 03 22:35:09.440206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:09.440175 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:17.440535 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:17.440500 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:17.453007 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:17.452973 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:17.786051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:17.785970 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:35:21.623032 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:35:21.622981 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:35:27.269691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.269655 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 22:35:27.274006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.273974 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.350461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.350429 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"premium-w4-32449d-kserve-self-signed-certs\"" Nov 03 22:35:27.367712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.367678 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67j6\" (UniqueName: \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.367851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.367764 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.367851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.367843 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.367927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.367902 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.368121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.368098 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.368197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.368147 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.402648 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.402624 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 22:35:27.469613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469583 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67j6\" (UniqueName: \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.469765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469655 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.469765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469694 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.469765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469736 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.469934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469808 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.469934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.469848 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.470190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.470123 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.470190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.470166 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.470370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.470198 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.472216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.472190 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.472536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.472520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.530321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.530259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67j6\" (UniqueName: \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\") pod \"premium-w4-32449d-kserve-5977cccc6b-vj666\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.587069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.587035 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:27.964139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:27.964115 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 22:35:27.965708 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:35:27.965685 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod199474a3_6246_434d_b6ba_52689cb281a0.slice/crio-204b9d3b239d6349e50583b4c5601e943f6481a3fcd6bacb90ea4a1a0b186501 WatchSource:0}: Error finding container 204b9d3b239d6349e50583b4c5601e943f6481a3fcd6bacb90ea4a1a0b186501: Status 404 returned error can't find the container with id 204b9d3b239d6349e50583b4c5601e943f6481a3fcd6bacb90ea4a1a0b186501 Nov 03 22:35:28.842311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:28.842277 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" event={"ID":"199474a3-6246-434d-b6ba-52689cb281a0","Type":"ContainerStarted","Data":"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f"} Nov 03 22:35:28.842311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:28.842318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" event={"ID":"199474a3-6246-434d-b6ba-52689cb281a0","Type":"ContainerStarted","Data":"204b9d3b239d6349e50583b4c5601e943f6481a3fcd6bacb90ea4a1a0b186501"} Nov 03 22:35:28.958051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:28.957955 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podStartSLOduration=1.9579373869999999 podStartE2EDuration="1.957937387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:28.957288082 +0000 UTC m=+802.694632309" watchObservedRunningTime="2026-11-03 22:35:28.957937387 +0000 UTC m=+802.695281613" Nov 03 22:35:29.587773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:29.587741 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:37.588065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:37.588025 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:37.601122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:37.601096 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:37.918051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:37.918027 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:35:42.218700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.218666 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 22:35:42.222824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.222809 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.280679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.280647 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 22:35:42.322655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.322631 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b64e2e\"/\"simulator-w0-b64e2e-kserve-self-signed-certs\"" Nov 03 22:35:42.326724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.326707 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b64e2e\"/\"kube-root-ca.crt\"" Nov 03 22:35:42.326824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.326739 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b64e2e\"/\"openshift-service-ca.crt\"" Nov 03 22:35:42.326824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.326739 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b64e2e\"/\"default-dockercfg-ffw7x\"" Nov 03 22:35:42.348524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.348642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348624 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.348698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348679 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.348748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348715 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.348803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqvh\" (UniqueName: \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.348840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.348832 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.449834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.449793 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.449862 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.449909 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.449940 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqvh\" (UniqueName: \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.449982 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.450082 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.450257 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.450425 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.450528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.450508 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.452467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.452444 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.452720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.452701 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.511089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.511031 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqvh\" (UniqueName: \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\") pod \"simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.535764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.535461 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:42.915375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.915347 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 22:35:42.917151 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:35:42.917126 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd0f9eb9_4ea5_4332_868e_9361e9e4448a.slice/crio-c4a01a9dc6970ebbe869763a2c9a5d51d1f2980804f707296d940da7ede93270 WatchSource:0}: Error finding container c4a01a9dc6970ebbe869763a2c9a5d51d1f2980804f707296d940da7ede93270: Status 404 returned error can't find the container with id c4a01a9dc6970ebbe869763a2c9a5d51d1f2980804f707296d940da7ede93270 Nov 03 22:35:42.940149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:42.940123 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" event={"ID":"bd0f9eb9-4ea5-4332-868e-9361e9e4448a","Type":"ContainerStarted","Data":"c4a01a9dc6970ebbe869763a2c9a5d51d1f2980804f707296d940da7ede93270"} Nov 03 22:35:43.948658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:43.948621 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" event={"ID":"bd0f9eb9-4ea5-4332-868e-9361e9e4448a","Type":"ContainerStarted","Data":"8e78f68df61efa25cf957a03668efa2e390dc65a54a1939505b4f049a14622ed"} Nov 03 22:35:44.066877 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:44.066817 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podStartSLOduration=2.066803619 podStartE2EDuration="2.066803619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:44.064766338 +0000 UTC m=+817.802110564" watchObservedRunningTime="2026-11-03 22:35:44.066803619 +0000 UTC m=+817.804147845" Nov 03 22:35:44.536065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:44.536034 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:51.623305 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:35:51.623267 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:35:52.536115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:52.536073 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:52.549064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:52.549041 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:35:53.013959 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:35:53.013930 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:36:21.620770 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:36:21.620697 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:36:23.245858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.245824 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 22:36:23.249583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.249563 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317516 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317562 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317651 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317711 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317770 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdd2\" (UniqueName: \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.317814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.317808 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.353464 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.353441 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"premium-w1-7afb1c-kserve-self-signed-certs\"" Nov 03 22:36:23.356549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.356519 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 22:36:23.419507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419474 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419518 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419547 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419592 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419612 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdd2\" (UniqueName: \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419635 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.419981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.419944 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.420280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.420257 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.420381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.420340 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.422267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.422240 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.422361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.422344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.480558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.480531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdd2\" (UniqueName: \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\") pod \"premium-w1-7afb1c-kserve-75cf954585-822p2\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.561098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.561035 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:23.985529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:23.985500 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 22:36:23.987576 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:36:23.987551 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode16565db_7bed_4f51_acf9_84afc0d7e603.slice/crio-23a7945101dc31181c7d26cd097ac637f50c93cf1d37365d1cfaf9488e5dbdf0 WatchSource:0}: Error finding container 23a7945101dc31181c7d26cd097ac637f50c93cf1d37365d1cfaf9488e5dbdf0: Status 404 returned error can't find the container with id 23a7945101dc31181c7d26cd097ac637f50c93cf1d37365d1cfaf9488e5dbdf0 Nov 03 22:36:24.230849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:24.230817 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" event={"ID":"e16565db-7bed-4f51-acf9-84afc0d7e603","Type":"ContainerStarted","Data":"51281819dad9db1de8800a05517d5bdfca6cfa22d0838bda1138f4facf4daa43"} Nov 03 22:36:24.231056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:24.230863 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" event={"ID":"e16565db-7bed-4f51-acf9-84afc0d7e603","Type":"ContainerStarted","Data":"23a7945101dc31181c7d26cd097ac637f50c93cf1d37365d1cfaf9488e5dbdf0"} Nov 03 22:36:24.351860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:24.351818 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podStartSLOduration=1.351804813 podStartE2EDuration="1.351804813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:24.349825291 +0000 UTC m=+858.087169539" watchObservedRunningTime="2026-11-03 22:36:24.351804813 +0000 UTC m=+858.089149039" Nov 03 22:36:24.561959 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:24.561925 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:33.561305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:33.561269 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:33.579977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:33.579951 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:34.307586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:34.307561 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:36:41.922257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:41.922216 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w"] Nov 03 22:36:41.926510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:41.926486 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:41.945110 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:41.945084 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:41.945269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:41.945250 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zjf\" (UniqueName: \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:41.945321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:41.945295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.028283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.028252 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:42.028454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.028299 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:36:42.032950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.032927 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w"] Nov 03 22:36:42.046462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.046438 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zjf\" (UniqueName: \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.046561 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.046487 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.046604 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.046571 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.047297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.047278 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.049044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.049028 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.107112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.107084 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zjf\" (UniqueName: \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\") pod \"maas-api-revoke-sub-42117eca-mq98w\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.240529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.240457 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:36:42.620145 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:42.620116 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w"] Nov 03 22:36:42.628189 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:36:42.628161 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb465d988_f46c_4d9e_83a8_a1e5f1da4eec.slice/crio-75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553 WatchSource:0}: Error finding container 75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553: Status 404 returned error can't find the container with id 75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553 Nov 03 22:36:43.353519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:43.353475 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" event={"ID":"b465d988-f46c-4d9e-83a8-a1e5f1da4eec","Type":"ContainerStarted","Data":"75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553"} Nov 03 22:36:51.623187 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:36:51.623145 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:36:59.512176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:59.512137 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" event={"ID":"b465d988-f46c-4d9e-83a8-a1e5f1da4eec","Type":"ContainerStarted","Data":"bce408617c31a07626f435588af23bd93cafe1995c33ca4e126016c4b7bb596e"} Nov 03 22:36:59.629349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:36:59.629295 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" podStartSLOduration=2.576909874 podStartE2EDuration="18.629282641s" totalImagesPullingTime="16.052372767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:59.6266936 +0000 UTC m=+893.364037824" watchObservedRunningTime="2026-11-03 22:36:59.629282641 +0000 UTC m=+893.366626865" Nov 03 22:37:00.519935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:00.519876 5051 generic.go:356] "Generic (PLEG): container finished" podID="b465d988-f46c-4d9e-83a8-a1e5f1da4eec" containerID="bce408617c31a07626f435588af23bd93cafe1995c33ca4e126016c4b7bb596e" exitCode=0 Nov 03 22:37:00.520474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:00.519962 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" event={"ID":"b465d988-f46c-4d9e-83a8-a1e5f1da4eec","Type":"ContainerDied","Data":"bce408617c31a07626f435588af23bd93cafe1995c33ca4e126016c4b7bb596e"} Nov 03 22:37:01.239833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.239800 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 22:37:01.243791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.243771 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.295884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.295860 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7540da-openshift-default-dockercfg-862xt\"" Nov 03 22:37:01.348091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.348059 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 22:37:01.382031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.381975 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382034 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382119 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382149 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382178 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382242 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr97d\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382341 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382363 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.382443 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.382398 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.483794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.483764 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.483952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.483808 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.483952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.483845 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.483952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.483874 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr97d\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.483965 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484020 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484090 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484168 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484193 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484415 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484473 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484665 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484797 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.484948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.484905 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.486392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.486365 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.486646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.486632 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.544812 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.544782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.545165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.544916 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr97d\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.557496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.557460 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:01.669925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.669899 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:37:01.787456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.787383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\") pod \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " Nov 03 22:37:01.787456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.787422 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\" (UniqueName: \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\") pod \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " Nov 03 22:37:01.788500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.788262 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\") pod \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\" (UID: \"b465d988-f46c-4d9e-83a8-a1e5f1da4eec\") " Nov 03 22:37:01.788500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.788422 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca" pod "b465d988-f46c-4d9e-83a8-a1e5f1da4eec" (UID: "b465d988-f46c-4d9e-83a8-a1e5f1da4eec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:01.788862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.788838 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:01.790606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.790580 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca" pod "b465d988-f46c-4d9e-83a8-a1e5f1da4eec" (UID: "b465d988-f46c-4d9e-83a8-a1e5f1da4eec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:01.791039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.791020 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf" pod "b465d988-f46c-4d9e-83a8-a1e5f1da4eec" (UID: "b465d988-f46c-4d9e-83a8-a1e5f1da4eec"). InnerVolumeSpecName "kube-api-access-h2zjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:01.890372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.890343 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:01.890372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:01.890373 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zjf\" (UniqueName: \"kubernetes.io/projected/b465d988-f46c-4d9e-83a8-a1e5f1da4eec-kube-api-access-h2zjf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:02.006263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.006218 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 22:37:02.007415 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:37:02.007384 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d68a831_daa6_43e3_b4b8_f559ca7b4902.slice/crio-d0fe964bebf7f5d735b005c8303b5772d7b92119b78c751d7f8b351b2d38a293 WatchSource:0}: Error finding container d0fe964bebf7f5d735b005c8303b5772d7b92119b78c751d7f8b351b2d38a293: Status 404 returned error can't find the container with id d0fe964bebf7f5d735b005c8303b5772d7b92119b78c751d7f8b351b2d38a293 Nov 03 22:37:02.009830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.009797 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:02.009919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.009860 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:02.009919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.009889 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:02.534445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.534419 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" Nov 03 22:37:02.534608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.534461 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w" event={"ID":"b465d988-f46c-4d9e-83a8-a1e5f1da4eec","Type":"ContainerDied","Data":"75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553"} Nov 03 22:37:02.534608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.534506 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cd2139f1dfbb8ab059909763886af4d6ae34621c6aeac84f23d27b73a10553" Nov 03 22:37:02.538198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.538172 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" event={"ID":"9d68a831-daa6-43e3-b4b8-f559ca7b4902","Type":"ContainerStarted","Data":"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161"} Nov 03 22:37:02.538298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.538220 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" event={"ID":"9d68a831-daa6-43e3-b4b8-f559ca7b4902","Type":"ContainerStarted","Data":"d0fe964bebf7f5d735b005c8303b5772d7b92119b78c751d7f8b351b2d38a293"} Nov 03 22:37:02.578137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.578101 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:02.709960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:02.709855 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podStartSLOduration=1.709840453 podStartE2EDuration="1.709840453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:02.708393778 +0000 UTC m=+896.445738002" watchObservedRunningTime="2026-11-03 22:37:02.709840453 +0000 UTC m=+896.447184757" Nov 03 22:37:03.581129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:03.580078 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:37:03.581129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:03.580133 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:37:04.560015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:04.558277 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:37:04.560015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:04.558332 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:37:05.561016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:05.560341 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:37:05.561016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:05.560396 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:37:06.570356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:06.570323 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:37:06.570918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:06.570889 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:37:06.589710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:06.589687 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:37:06.590843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:06.590824 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:37:07.565014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:07.561076 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 22:37:07.565014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:07.561147 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 22:37:08.562575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:08.562543 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:08.563156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:08.563130 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:08.564055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:08.564034 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 22:37:21.625787 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:21.625740 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:37:28.923947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.923913 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 22:37:28.924772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.924752 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b465d988-f46c-4d9e-83a8-a1e5f1da4eec" containerName="revoke-keys" Nov 03 22:37:28.924816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.924776 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b465d988-f46c-4d9e-83a8-a1e5f1da4eec" containerName="revoke-keys" Nov 03 22:37:28.924947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.924936 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b465d988-f46c-4d9e-83a8-a1e5f1da4eec" containerName="revoke-keys" Nov 03 22:37:28.928793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.928776 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:28.985958 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:28.985925 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 22:37:29.031747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.031711 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"kube-root-ca.crt\"" Nov 03 22:37:29.031984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.031795 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"openshift-service-ca.crt\"" Nov 03 22:37:29.031984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.031846 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"premium-w3-1d31f8-kserve-self-signed-certs\"" Nov 03 22:37:29.031984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.031968 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"default-dockercfg-l54sw\"" Nov 03 22:37:29.041555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041531 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.041654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.041654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041584 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.041721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fwt\" (UniqueName: \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.041773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041755 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.041820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.041809 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143472 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143509 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143561 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fwt\" (UniqueName: \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143605 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143636 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.143962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143936 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.144055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.143969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.144122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.144092 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.146035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.146014 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.146209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.146194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.204198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.204129 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fwt\" (UniqueName: \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\") pod \"premium-w3-1d31f8-kserve-67769f7766-5prkw\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.243089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.243056 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:29.662482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.662458 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 22:37:29.664478 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:37:29.664451 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod756749a1_27bd_499b_8eff_ddc032b61f43.slice/crio-40710374fab5591ff5d996665ab0ed6e2c4dd3a9b640fdcc031ca5e108094ec4 WatchSource:0}: Error finding container 40710374fab5591ff5d996665ab0ed6e2c4dd3a9b640fdcc031ca5e108094ec4: Status 404 returned error can't find the container with id 40710374fab5591ff5d996665ab0ed6e2c4dd3a9b640fdcc031ca5e108094ec4 Nov 03 22:37:29.766308 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.766267 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" event={"ID":"756749a1-27bd-499b-8eff-ddc032b61f43","Type":"ContainerStarted","Data":"76bf8e34ed0d8dd7e6b2fe1a5c939d5ad4b27bfd6e2c2d11ae58b28f01f73f55"} Nov 03 22:37:29.766481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.766327 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" event={"ID":"756749a1-27bd-499b-8eff-ddc032b61f43","Type":"ContainerStarted","Data":"40710374fab5591ff5d996665ab0ed6e2c4dd3a9b640fdcc031ca5e108094ec4"} Nov 03 22:37:29.890371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:29.890328 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podStartSLOduration=1.890315437 podStartE2EDuration="1.890315437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:29.886259332 +0000 UTC m=+923.623603557" watchObservedRunningTime="2026-11-03 22:37:29.890315437 +0000 UTC m=+923.627659661" Nov 03 22:37:30.244085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:30.244057 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:31.445409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.445380 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2"] Nov 03 22:37:31.453540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.453520 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.506911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.506882 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2"] Nov 03 22:37:31.552520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.552497 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:31.552644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.552564 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:37:31.581816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.581791 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.581932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.581853 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.581984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.581965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68sq4\" (UniqueName: \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.682860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.682840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.682960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.682885 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68sq4\" (UniqueName: \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.683038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.682966 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.683513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.683492 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.685283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.685263 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.743486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.743431 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68sq4\" (UniqueName: \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\") pod \"maas-api-revoke-sub-ee382c38-k75w2\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:31.768274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:31.768253 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:32.148520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:32.148476 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2"] Nov 03 22:37:32.151389 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:37:32.151360 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9563d519_9066_4803_8974_cc58ef958560.slice/crio-06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34 WatchSource:0}: Error finding container 06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34: Status 404 returned error can't find the container with id 06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34 Nov 03 22:37:32.785665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:32.785600 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" event={"ID":"9563d519-9066-4803-8974-cc58ef958560","Type":"ContainerStarted","Data":"8abfd020c53924715eb6a5d1d63cf7d0872c1f4e6f5f864683c6cc74bfe8a8d9"} Nov 03 22:37:32.785665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:32.785647 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" event={"ID":"9563d519-9066-4803-8974-cc58ef958560","Type":"ContainerStarted","Data":"06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34"} Nov 03 22:37:32.903662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:32.903620 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" podStartSLOduration=1.903606356 podStartE2EDuration="1.903606356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:32.900195445 +0000 UTC m=+926.637539666" watchObservedRunningTime="2026-11-03 22:37:32.903606356 +0000 UTC m=+926.640950579" Nov 03 22:37:33.792682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:33.792648 5051 generic.go:356] "Generic (PLEG): container finished" podID="9563d519-9066-4803-8974-cc58ef958560" containerID="8abfd020c53924715eb6a5d1d63cf7d0872c1f4e6f5f864683c6cc74bfe8a8d9" exitCode=0 Nov 03 22:37:33.793091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:33.792702 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" event={"ID":"9563d519-9066-4803-8974-cc58ef958560","Type":"ContainerDied","Data":"8abfd020c53924715eb6a5d1d63cf7d0872c1f4e6f5f864683c6cc74bfe8a8d9"} Nov 03 22:37:34.947487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:34.947460 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:35.025930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.025906 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\" (UniqueName: \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\") pod \"9563d519-9066-4803-8974-cc58ef958560\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " Nov 03 22:37:35.026123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.025953 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\") pod \"9563d519-9066-4803-8974-cc58ef958560\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " Nov 03 22:37:35.026123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.026053 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\") pod \"9563d519-9066-4803-8974-cc58ef958560\" (UID: \"9563d519-9066-4803-8974-cc58ef958560\") " Nov 03 22:37:35.026259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.026235 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca" pod "9563d519-9066-4803-8974-cc58ef958560" (UID: "9563d519-9066-4803-8974-cc58ef958560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:35.026520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.026499 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9563d519-9066-4803-8974-cc58ef958560-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.028030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.028005 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4" pod "9563d519-9066-4803-8974-cc58ef958560" (UID: "9563d519-9066-4803-8974-cc58ef958560"). InnerVolumeSpecName "kube-api-access-68sq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:35.028192 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.028176 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca" pod "9563d519-9066-4803-8974-cc58ef958560" (UID: "9563d519-9066-4803-8974-cc58ef958560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:35.128042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.127977 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68sq4\" (UniqueName: \"kubernetes.io/projected/9563d519-9066-4803-8974-cc58ef958560-kube-api-access-68sq4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.128042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.128041 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9563d519-9066-4803-8974-cc58ef958560-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.805636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.805612 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" Nov 03 22:37:35.805839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.805647 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2" event={"ID":"9563d519-9066-4803-8974-cc58ef958560","Type":"ContainerDied","Data":"06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34"} Nov 03 22:37:35.805839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:35.805689 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e3e3d89dc705661b4eb65ed9e503f82df1f01440ae67bc6871e060156a5e34" Nov 03 22:37:39.243438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:39.243405 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:39.255500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:39.255477 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:39.840635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:39.840612 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:37:47.102076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.101983 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn"] Nov 03 22:37:47.102587 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.102573 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9563d519-9066-4803-8974-cc58ef958560" containerName="revoke-keys" Nov 03 22:37:47.102626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.102588 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9563d519-9066-4803-8974-cc58ef958560" containerName="revoke-keys" Nov 03 22:37:47.102705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.102696 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9563d519-9066-4803-8974-cc58ef958560" containerName="revoke-keys" Nov 03 22:37:47.107154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.107138 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.163002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.162958 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn"] Nov 03 22:37:47.163775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.163749 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.163876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.163788 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.163876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.163844 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmx8\" (UniqueName: \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.208528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.208506 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:47.208635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.208547 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:37:47.264771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.264747 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.264880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.264784 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.264880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.264817 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmx8\" (UniqueName: \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.265585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.265551 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.267442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.267420 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.327589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.327559 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmx8\" (UniqueName: \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\") pod \"maas-api-revoke-sub-82d39c3d-f4nkn\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.420934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.420861 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:47.799288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.799260 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn"] Nov 03 22:37:47.801026 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:37:47.800978 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a40e3c_d61c_4346_b9e4_4c00fcb773c9.slice/crio-6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442 WatchSource:0}: Error finding container 6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442: Status 404 returned error can't find the container with id 6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442 Nov 03 22:37:47.894504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:47.894475 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" event={"ID":"33a40e3c-d61c-4346-b9e4-4c00fcb773c9","Type":"ContainerStarted","Data":"6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442"} Nov 03 22:37:48.901602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:48.901534 5051 generic.go:356] "Generic (PLEG): container finished" podID="33a40e3c-d61c-4346-b9e4-4c00fcb773c9" containerID="f9600a353f35e2a4b423717990c616bc7de79c92a382b6f28a77d689d2d22356" exitCode=0 Nov 03 22:37:48.901602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:48.901573 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" event={"ID":"33a40e3c-d61c-4346-b9e4-4c00fcb773c9","Type":"ContainerDied","Data":"f9600a353f35e2a4b423717990c616bc7de79c92a382b6f28a77d689d2d22356"} Nov 03 22:37:50.074266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.074239 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:50.190957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.190877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\" (UniqueName: \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\") pod \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " Nov 03 22:37:50.190957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.190938 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\") pod \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " Nov 03 22:37:50.191201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.191078 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\") pod \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\" (UID: \"33a40e3c-d61c-4346-b9e4-4c00fcb773c9\") " Nov 03 22:37:50.191693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.191668 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca" pod "33a40e3c-d61c-4346-b9e4-4c00fcb773c9" (UID: "33a40e3c-d61c-4346-b9e4-4c00fcb773c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:50.193847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.193821 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca" pod "33a40e3c-d61c-4346-b9e4-4c00fcb773c9" (UID: "33a40e3c-d61c-4346-b9e4-4c00fcb773c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:50.193847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.193829 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8" pod "33a40e3c-d61c-4346-b9e4-4c00fcb773c9" (UID: "33a40e3c-d61c-4346-b9e4-4c00fcb773c9"). InnerVolumeSpecName "kube-api-access-krmx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:50.293176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.293146 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.293279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.293178 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.293279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.293197 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmx8\" (UniqueName: \"kubernetes.io/projected/33a40e3c-d61c-4346-b9e4-4c00fcb773c9-kube-api-access-krmx8\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:50.456449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.456391 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:37:50.456615 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.456592 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" containerID="cri-o://2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304" gracePeriod=30 Nov 03 22:37:50.507943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.507911 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:37:50.508164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.508141 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" containerID="cri-o://3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7" gracePeriod=30 Nov 03 22:37:50.512660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.512637 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:37:50.512840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.512820 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" containerID="cri-o://1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865" gracePeriod=30 Nov 03 22:37:50.516765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.516741 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:37:50.516946 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.516926 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" containerID="cri-o://88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d" gracePeriod=30 Nov 03 22:37:50.914711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.914684 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" Nov 03 22:37:50.914880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.914684 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn" event={"ID":"33a40e3c-d61c-4346-b9e4-4c00fcb773c9","Type":"ContainerDied","Data":"6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442"} Nov 03 22:37:50.914880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:50.914836 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6015c81d75ba5766a54d5b1a4cbcaa20e76bd1c9929657e2812a3f1c1f732442" Nov 03 22:37:51.624559 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:51.624526 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:37:54.127979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:54.127938 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jr4dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:37:54.128479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:54.128001 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:37:55.802130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.802105 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:37:55.865689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.865660 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.865820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.865726 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.865897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.865882 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.865960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.865924 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.865960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.865955 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.866277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866252 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.866349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866306 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.866349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866319 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.866455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866350 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.866455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866406 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\") pod \"d1e7683c-3605-4397-8157-074947210385\" (UID: \"d1e7683c-3605-4397-8157-074947210385\") " Nov 03 22:37:55.866619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866574 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.866684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.867182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866935 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.867182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866959 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.867182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.866974 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.867182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.867149 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:55.873251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.873227 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.873378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.873361 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:55.873475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.873452 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "kube-api-access-4j4r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.873573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.873554 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.873673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.873657 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy" pod "d1e7683c-3605-4397-8157-074947210385" (UID: "d1e7683c-3605-4397-8157-074947210385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.879173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.879152 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:37:55.881699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.881483 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:37:55.885069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.883605 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:37:55.951208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.951185 5051 generic.go:356] "Generic (PLEG): container finished" podID="068095c9-59cb-4e19-8183-784135e57e88" containerID="88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d" exitCode=0 Nov 03 22:37:55.951334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.951314 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" Nov 03 22:37:55.951377 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.951325 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" event={"ID":"068095c9-59cb-4e19-8183-784135e57e88","Type":"ContainerDied","Data":"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d"} Nov 03 22:37:55.951420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.951402 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr" event={"ID":"068095c9-59cb-4e19-8183-784135e57e88","Type":"ContainerDied","Data":"f2eaf0b693280ad8f1f642128f09948b01e0a048b5894974d5705c1efa060136"} Nov 03 22:37:55.951484 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.951429 5051 scope.go:122] "RemoveContainer" containerID="88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d" Nov 03 22:37:55.952723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.952704 5051 generic.go:356] "Generic (PLEG): container finished" podID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerID="3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7" exitCode=0 Nov 03 22:37:55.952818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.952752 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" Nov 03 22:37:55.952818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.952773 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" event={"ID":"df6c0a15-936f-4ed4-b31b-02107af53d62","Type":"ContainerDied","Data":"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7"} Nov 03 22:37:55.952818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.952808 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh" event={"ID":"df6c0a15-936f-4ed4-b31b-02107af53d62","Type":"ContainerDied","Data":"b575105ff281a5efbfab184ee08b26c33a737740b25f127556b918c4ff9b06e0"} Nov 03 22:37:55.954234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.954210 5051 generic.go:356] "Generic (PLEG): container finished" podID="d1e7683c-3605-4397-8157-074947210385" containerID="2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304" exitCode=0 Nov 03 22:37:55.954329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.954253 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" event={"ID":"d1e7683c-3605-4397-8157-074947210385","Type":"ContainerDied","Data":"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304"} Nov 03 22:37:55.954329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.954271 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" Nov 03 22:37:55.954329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.954291 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb" event={"ID":"d1e7683c-3605-4397-8157-074947210385","Type":"ContainerDied","Data":"b3c2a7f4310ff3409ce19ea52f5240e8b4cf691607058f8afb6a325cf1975c2b"} Nov 03 22:37:55.955504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.955487 5051 generic.go:356] "Generic (PLEG): container finished" podID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerID="1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865" exitCode=0 Nov 03 22:37:55.955596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.955527 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" Nov 03 22:37:55.955596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.955543 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" event={"ID":"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef","Type":"ContainerDied","Data":"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865"} Nov 03 22:37:55.955596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.955566 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd" event={"ID":"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef","Type":"ContainerDied","Data":"c10c51644fd1e00d1b2d579efc650218ee8a194e51b9d8dae22afa418af8b9ae"} Nov 03 22:37:55.962444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.962430 5051 scope.go:122] "RemoveContainer" containerID="88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d" Nov 03 22:37:55.962711 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:55.962687 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d\": container with ID starting with 88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d not found: ID does not exist" containerID="88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d" Nov 03 22:37:55.962778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.962722 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d"} err="failed to get container status \"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d\": rpc error: code = NotFound desc = could not find container \"88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d\": container with ID starting with 88261329f4219238ad8e9931d1fba82b981f5a1c9e797373e978c2d497c24f9d not found: ID does not exist" Nov 03 22:37:55.962778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.962745 5051 scope.go:122] "RemoveContainer" containerID="3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7" Nov 03 22:37:55.967811 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967791 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.967901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967837 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.967901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967859 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.967901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967893 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967937 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.967968 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968054 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968087 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968136 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968174 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968204 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968206 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968234 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968256 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.968596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968412 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968460 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968500 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968539 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968591 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968643 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968679 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.968826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968729 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968788 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968836 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.968967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968914 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.968967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968953 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\") pod \"df6c0a15-936f-4ed4-b31b-02107af53d62\" (UID: \"df6c0a15-936f-4ed4-b31b-02107af53d62\") " Nov 03 22:37:55.969143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.969014 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\") pod \"068095c9-59cb-4e19-8183-784135e57e88\" (UID: \"068095c9-59cb-4e19-8183-784135e57e88\") " Nov 03 22:37:55.969143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.969055 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\") pod \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\" (UID: \"34f81a16-6fe8-4468-8f5d-4c6dbeba3fef\") " Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970273 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970300 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e7683c-3605-4397-8157-074947210385-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970314 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970328 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e7683c-3605-4397-8157-074947210385-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970340 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970352 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970366 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4r5\" (UniqueName: \"kubernetes.io/projected/d1e7683c-3605-4397-8157-074947210385-kube-api-access-4j4r5\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.970380 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e7683c-3605-4397-8157-074947210385-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968414 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.968663 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.969642 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.969824 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.971853 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.972593 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "kube-api-access-lvjp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.972862 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.972906 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973410 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973627 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973655 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973779 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973795 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.973837 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974306 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974323 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974415 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974417 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974439 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "kube-api-access-rwqj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.975175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.974444 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:55.976594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.976575 5051 scope.go:122] "RemoveContainer" containerID="3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7" Nov 03 22:37:55.976876 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:55.976857 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7\": container with ID starting with 3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7 not found: ID does not exist" containerID="3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7" Nov 03 22:37:55.976918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.976886 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7"} err="failed to get container status \"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7\": rpc error: code = NotFound desc = could not find container \"3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7\": container with ID starting with 3a4301c22ce795122d223278dbe0ea04771776a3ec029351b26e82760a500ae7 not found: ID does not exist" Nov 03 22:37:55.976918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.976907 5051 scope.go:122] "RemoveContainer" containerID="2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304" Nov 03 22:37:55.984678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.984613 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token" pod "068095c9-59cb-4e19-8183-784135e57e88" (UID: "068095c9-59cb-4e19-8183-784135e57e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.984925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.984898 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "kube-api-access-d4sq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.985138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.985111 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy" pod "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" (UID: "34f81a16-6fe8-4468-8f5d-4c6dbeba3fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.987880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.987799 5051 scope.go:122] "RemoveContainer" containerID="2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304" Nov 03 22:37:55.988315 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:55.988233 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304\": container with ID starting with 2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304 not found: ID does not exist" containerID="2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304" Nov 03 22:37:55.988315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.988272 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304"} err="failed to get container status \"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304\": rpc error: code = NotFound desc = could not find container \"2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304\": container with ID starting with 2f34f89267b398985d4171e92569196d2906aefec61b5ecad6dd0f753669d304 not found: ID does not exist" Nov 03 22:37:55.988315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.988295 5051 scope.go:122] "RemoveContainer" containerID="1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865" Nov 03 22:37:55.994380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.994350 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:55.994512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.994489 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token" pod "df6c0a15-936f-4ed4-b31b-02107af53d62" (UID: "df6c0a15-936f-4ed4-b31b-02107af53d62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:55.999589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.999574 5051 scope.go:122] "RemoveContainer" containerID="1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865" Nov 03 22:37:55.999888 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:37:55.999865 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865\": container with ID starting with 1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865 not found: ID does not exist" containerID="1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865" Nov 03 22:37:56.000014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:55.999893 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865"} err="failed to get container status \"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865\": rpc error: code = NotFound desc = could not find container \"1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865\": container with ID starting with 1a30ebd479d28c25a930fbd2c3b2433aa1a791bdb3c1b2a2750dbf572f8bb865 not found: ID does not exist" Nov 03 22:37:56.071532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071478 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071500 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjp7\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-kube-api-access-lvjp7\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071509 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071518 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068095c9-59cb-4e19-8183-784135e57e88-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071527 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071536 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071545 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqj5\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-kube-api-access-rwqj5\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071554 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071562 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071570 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068095c9-59cb-4e19-8183-784135e57e88-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071578 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071586 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071594 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071603 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071610 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071618 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071627 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071636 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068095c9-59cb-4e19-8183-784135e57e88-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071643 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df6c0a15-936f-4ed4-b31b-02107af53d62-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071651 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071658 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df6c0a15-936f-4ed4-b31b-02107af53d62-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071666 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df6c0a15-936f-4ed4-b31b-02107af53d62-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071674 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068095c9-59cb-4e19-8183-784135e57e88-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071681 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sq6\" (UniqueName: \"kubernetes.io/projected/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-kube-api-access-d4sq6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.071743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.071690 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:37:56.338392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.338322 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:37:56.340285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.340264 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fsjfb"] Nov 03 22:37:56.407868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.407845 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e7683c-3605-4397-8157-074947210385" path="/var/lib/kubelet/pods/d1e7683c-3605-4397-8157-074947210385/volumes" Nov 03 22:37:56.575479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.575441 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:37:56.579946 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.579919 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9gvgr"] Nov 03 22:37:56.688681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.688652 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:37:56.694619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.694593 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn8gd"] Nov 03 22:37:56.805366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.805341 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:37:56.808638 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:56.808613 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jr4dh"] Nov 03 22:37:58.118233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118204 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4"] Nov 03 22:37:58.118695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118681 5051 cpu_manager.go:414] "containerMap: removing container" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" Nov 03 22:37:58.118747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118696 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" Nov 03 22:37:58.118747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118706 5051 cpu_manager.go:414] "containerMap: removing container" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" Nov 03 22:37:58.118747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118713 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" Nov 03 22:37:58.118747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118733 5051 cpu_manager.go:414] "containerMap: removing container" podUID="33a40e3c-d61c-4346-b9e4-4c00fcb773c9" containerName="revoke-keys" Nov 03 22:37:58.118747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118738 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a40e3c-d61c-4346-b9e4-4c00fcb773c9" containerName="revoke-keys" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118749 5051 cpu_manager.go:414] "containerMap: removing container" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118754 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118769 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118773 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118856 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118867 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e7683c-3605-4397-8157-074947210385" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118877 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118882 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="068095c9-59cb-4e19-8183-784135e57e88" containerName="istio-proxy" Nov 03 22:37:58.118898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.118887 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a40e3c-d61c-4346-b9e4-4c00fcb773c9" containerName="revoke-keys" Nov 03 22:37:58.123797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.123779 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.179931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.179910 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4"] Nov 03 22:37:58.192255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.192225 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lz6f\" (UniqueName: \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.192370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.192343 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.192435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.192378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.225423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.225406 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:58.225507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.225473 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:37:58.293533 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.293509 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.293656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.293625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lz6f\" (UniqueName: \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.293716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.293701 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.294329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.294309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.296207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.296186 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.354331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.354309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lz6f\" (UniqueName: \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\") pod \"maas-api-revoke-sub-ff96fe4c-257f4\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.407429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.407377 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068095c9-59cb-4e19-8183-784135e57e88" path="/var/lib/kubelet/pods/068095c9-59cb-4e19-8183-784135e57e88/volumes" Nov 03 22:37:58.408298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.408270 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f81a16-6fe8-4468-8f5d-4c6dbeba3fef" path="/var/lib/kubelet/pods/34f81a16-6fe8-4468-8f5d-4c6dbeba3fef/volumes" Nov 03 22:37:58.408951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.408933 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6c0a15-936f-4ed4-b31b-02107af53d62" path="/var/lib/kubelet/pods/df6c0a15-936f-4ed4-b31b-02107af53d62/volumes" Nov 03 22:37:58.436942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.436919 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:37:58.818823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.818801 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4"] Nov 03 22:37:58.820574 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:37:58.820534 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a077ff_dc68_4334_afa9_8fc5d25e30c1.slice/crio-4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc WatchSource:0}: Error finding container 4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc: Status 404 returned error can't find the container with id 4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc Nov 03 22:37:58.980350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:58.980324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" event={"ID":"62a077ff-dc68-4334-afa9-8fc5d25e30c1","Type":"ContainerStarted","Data":"4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc"} Nov 03 22:37:59.988500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:59.988413 5051 generic.go:356] "Generic (PLEG): container finished" podID="62a077ff-dc68-4334-afa9-8fc5d25e30c1" containerID="5f26f959bfe367b491ca48f367b3981b36c81bb82ecba124d92e48e56a428330" exitCode=0 Nov 03 22:37:59.989083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:37:59.988494 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" event={"ID":"62a077ff-dc68-4334-afa9-8fc5d25e30c1","Type":"ContainerDied","Data":"5f26f959bfe367b491ca48f367b3981b36c81bb82ecba124d92e48e56a428330"} Nov 03 22:38:00.686884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.686853 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:38:00.691760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.691738 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.744048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.744020 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-32449d-openshift-default-dockercfg-rc6q5\"" Nov 03 22:38:00.796687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.796659 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:38:00.821293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821264 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821434 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821473 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821592 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821573 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821613 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821734 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821720 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5p7\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821772 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.821847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.821801 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923028 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5p7\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923073 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923101 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923155 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923242 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923276 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923302 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923353 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923393 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923560 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923835 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923861 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923839 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.923901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.923940 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.925600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.925577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:00.925878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:00.925852 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:01.018983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.018951 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5p7\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:01.019458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.019432 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:01.143638 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.143618 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:38:01.226875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.226845 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\") pod \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " Nov 03 22:38:01.227016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.226905 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\" (UniqueName: \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\") pod \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " Nov 03 22:38:01.227118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.227099 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\") pod \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\" (UID: \"62a077ff-dc68-4334-afa9-8fc5d25e30c1\") " Nov 03 22:38:01.227188 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.227166 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca" pod "62a077ff-dc68-4334-afa9-8fc5d25e30c1" (UID: "62a077ff-dc68-4334-afa9-8fc5d25e30c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:01.227494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.227477 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:01.228954 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.228937 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f" pod "62a077ff-dc68-4334-afa9-8fc5d25e30c1" (UID: "62a077ff-dc68-4334-afa9-8fc5d25e30c1"). InnerVolumeSpecName "kube-api-access-8lz6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:01.229141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.229121 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca" pod "62a077ff-dc68-4334-afa9-8fc5d25e30c1" (UID: "62a077ff-dc68-4334-afa9-8fc5d25e30c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:01.308433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.308352 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:01.328462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.328429 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lz6f\" (UniqueName: \"kubernetes.io/projected/62a077ff-dc68-4334-afa9-8fc5d25e30c1-kube-api-access-8lz6f\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:01.328582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.328473 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a077ff-dc68-4334-afa9-8fc5d25e30c1-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:01.741092 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:01.741065 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3aa3a7_defd_4022_9551_abf2611d246c.slice/crio-e3666408f8033363cabdc5bf0416c8167413b98b309ed726abfe327e7d8fdfb3 WatchSource:0}: Error finding container e3666408f8033363cabdc5bf0416c8167413b98b309ed726abfe327e7d8fdfb3: Status 404 returned error can't find the container with id e3666408f8033363cabdc5bf0416c8167413b98b309ed726abfe327e7d8fdfb3 Nov 03 22:38:01.741330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.741306 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:38:01.743301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.743269 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:01.743415 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.743326 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:01.743415 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:01.743377 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:02.003269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.003200 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" Nov 03 22:38:02.003427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.003401 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4" event={"ID":"62a077ff-dc68-4334-afa9-8fc5d25e30c1","Type":"ContainerDied","Data":"4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc"} Nov 03 22:38:02.003492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.003452 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eef20676c3af8478c57241b5f6918c229186e5e34ce65c8d6f5400b5e1e6ddc" Nov 03 22:38:02.005445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.005420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" event={"ID":"7a3aa3a7-defd-4022-9551-abf2611d246c","Type":"ContainerStarted","Data":"a13ddd8012c8e46d0b7d7d82bb26444a11702498c736db5d590dcee6b1bbbe41"} Nov 03 22:38:02.005555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.005478 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" event={"ID":"7a3aa3a7-defd-4022-9551-abf2611d246c","Type":"ContainerStarted","Data":"e3666408f8033363cabdc5bf0416c8167413b98b309ed726abfe327e7d8fdfb3"} Nov 03 22:38:02.128899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.128843 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podStartSLOduration=2.128826689 podStartE2EDuration="2.128826689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:02.1255201 +0000 UTC m=+955.862864324" watchObservedRunningTime="2026-11-03 22:38:02.128826689 +0000 UTC m=+955.866170914" Nov 03 22:38:02.309609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.309519 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:02.311142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.311118 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:02.311256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.311163 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:02.638078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.638046 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 22:38:02.638567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.638555 5051 cpu_manager.go:414] "containerMap: removing container" podUID="62a077ff-dc68-4334-afa9-8fc5d25e30c1" containerName="revoke-keys" Nov 03 22:38:02.638617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.638568 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a077ff-dc68-4334-afa9-8fc5d25e30c1" containerName="revoke-keys" Nov 03 22:38:02.638695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.638682 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a077ff-dc68-4334-afa9-8fc5d25e30c1" containerName="revoke-keys" Nov 03 22:38:02.642208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.642191 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.694143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.694113 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1d31f8-openshift-default-dockercfg-wj4vg\"" Nov 03 22:38:02.745498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745472 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745502 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745587 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745612 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745636 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745708 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf54h\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745749 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.745796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.745782 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.750439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.750415 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 22:38:02.792262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.792233 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv"] Nov 03 22:38:02.796695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.796667 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:02.846555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.846531 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf54h\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.846680 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.846579 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.846748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.846728 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.846848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.846834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.846929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.846862 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847033 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847068 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847125 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847164 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847257 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847416 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847473 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.847730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.847712 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.848552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.848531 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:02.848740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.848702 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:02.849351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.849329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.849625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.849608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.889858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.889770 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv"] Nov 03 22:38:02.907683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.907654 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf54h\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.908676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.908650 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:02.948579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.948550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96hj\" (UniqueName: \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:02.948702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.948685 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:02.948750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.948739 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:02.955311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:02.955291 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:03.057102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.052731 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96hj\" (UniqueName: \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.057102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.052880 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.057102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.052944 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.057102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.053745 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.102019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.097700 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.115631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.115604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96hj\" (UniqueName: \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\") pod \"maas-api-revoke-sub-e501ba38-7swkv\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.309456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.309360 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:03.309456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.309425 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:03.411705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.411677 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:03.422856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.422827 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 22:38:03.424101 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:03.424074 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0908daf7_a76e_463d_bfc7_7a7ea80806fa.slice/crio-0b057b93b8ecc3c126ca09f01d61f46b9f5c89492c3ab24bb9e7a140953b5001 WatchSource:0}: Error finding container 0b057b93b8ecc3c126ca09f01d61f46b9f5c89492c3ab24bb9e7a140953b5001: Status 404 returned error can't find the container with id 0b057b93b8ecc3c126ca09f01d61f46b9f5c89492c3ab24bb9e7a140953b5001 Nov 03 22:38:03.426791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.426756 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:03.426901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.426821 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:03.426901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.426850 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:38:03.821670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:03.821636 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv"] Nov 03 22:38:03.825410 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:03.825372 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82957650_f195_4666_a95d_228727c46d47.slice/crio-6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0 WatchSource:0}: Error finding container 6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0: Status 404 returned error can't find the container with id 6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0 Nov 03 22:38:04.023325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.023288 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" event={"ID":"0908daf7-a76e-463d-bfc7-7a7ea80806fa","Type":"ContainerStarted","Data":"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a"} Nov 03 22:38:04.023580 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.023564 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" event={"ID":"0908daf7-a76e-463d-bfc7-7a7ea80806fa","Type":"ContainerStarted","Data":"0b057b93b8ecc3c126ca09f01d61f46b9f5c89492c3ab24bb9e7a140953b5001"} Nov 03 22:38:04.026357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.026279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" event={"ID":"82957650-f195-4666-a95d-228727c46d47","Type":"ContainerStarted","Data":"6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0"} Nov 03 22:38:04.146284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.146232 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podStartSLOduration=2.14621806 podStartE2EDuration="2.14621806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:04.143365441 +0000 UTC m=+957.880709666" watchObservedRunningTime="2026-11-03 22:38:04.14621806 +0000 UTC m=+957.883562363" Nov 03 22:38:04.311354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.311316 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:04.311817 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.311375 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:04.955620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.955585 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:04.971013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.960118 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:38:04.971013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:04.960172 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:38:05.034768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:05.034681 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" event={"ID":"82957650-f195-4666-a95d-228727c46d47","Type":"ContainerStarted","Data":"d5e5e218eaf2c8a85eb561f7f9a4e465f4aecb68245ef20fb70f5a9756f8dc7b"} Nov 03 22:38:05.309072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:05.308967 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:05.309315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:05.309288 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:05.955852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:05.955825 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:38:05.974092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:05.974052 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:38:06.044204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.044168 5051 generic.go:356] "Generic (PLEG): container finished" podID="82957650-f195-4666-a95d-228727c46d47" containerID="d5e5e218eaf2c8a85eb561f7f9a4e465f4aecb68245ef20fb70f5a9756f8dc7b" exitCode=0 Nov 03 22:38:06.044361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.044251 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" event={"ID":"82957650-f195-4666-a95d-228727c46d47","Type":"ContainerDied","Data":"d5e5e218eaf2c8a85eb561f7f9a4e465f4aecb68245ef20fb70f5a9756f8dc7b"} Nov 03 22:38:06.318089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.317353 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:06.318089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.317420 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:06.427711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.427551 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:06.434183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.434159 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\" (UniqueName: \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\") pod \"82957650-f195-4666-a95d-228727c46d47\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " Nov 03 22:38:06.434332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.434254 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\") pod \"82957650-f195-4666-a95d-228727c46d47\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " Nov 03 22:38:06.434405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.434358 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\") pod \"82957650-f195-4666-a95d-228727c46d47\" (UID: \"82957650-f195-4666-a95d-228727c46d47\") " Nov 03 22:38:06.434700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.434679 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca" pod "82957650-f195-4666-a95d-228727c46d47" (UID: "82957650-f195-4666-a95d-228727c46d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:06.434847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.434830 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82957650-f195-4666-a95d-228727c46d47-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.436280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.436257 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca" pod "82957650-f195-4666-a95d-228727c46d47" (UID: "82957650-f195-4666-a95d-228727c46d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:06.441111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.441090 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj" pod "82957650-f195-4666-a95d-228727c46d47" (UID: "82957650-f195-4666-a95d-228727c46d47"). InnerVolumeSpecName "kube-api-access-k96hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:06.535890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.535861 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96hj\" (UniqueName: \"kubernetes.io/projected/82957650-f195-4666-a95d-228727c46d47-kube-api-access-k96hj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.535890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.535884 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82957650-f195-4666-a95d-228727c46d47-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:06.965014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.960086 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:38:06.965014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:06.960140 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:38:07.052578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.052543 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" Nov 03 22:38:07.052771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.052655 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv" event={"ID":"82957650-f195-4666-a95d-228727c46d47","Type":"ContainerDied","Data":"6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0"} Nov 03 22:38:07.052771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.052696 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc4e09945762a64b714c27f75671ce0e6bb75d844dfe4007db5ba971ebe20e0" Nov 03 22:38:07.309682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.309227 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:38:07.309682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.309282 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 22:38:07.957015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.956262 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:38:07.957015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:07.956319 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:38:08.341371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:08.341284 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:08.341904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:08.341764 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:08.342856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:08.342837 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:38:08.956015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:08.955962 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:38:08.956217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:08.956063 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:38:10.029368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:10.029321 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:10.029846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:10.029681 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:10.030386 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:10.030356 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 22:38:18.319870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.319835 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk"] Nov 03 22:38:18.320421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.320406 5051 cpu_manager.go:414] "containerMap: removing container" podUID="82957650-f195-4666-a95d-228727c46d47" containerName="revoke-keys" Nov 03 22:38:18.320467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.320422 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82957650-f195-4666-a95d-228727c46d47" containerName="revoke-keys" Nov 03 22:38:18.320544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.320535 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="82957650-f195-4666-a95d-228727c46d47" containerName="revoke-keys" Nov 03 22:38:18.323960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.323945 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.380573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.380542 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk"] Nov 03 22:38:18.384945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.384914 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.385120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.384964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8mn\" (UniqueName: \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.385192 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.385160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.425903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.425877 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:18.426096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.425915 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:18.486537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.486500 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.486677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.486646 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.486677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.486672 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8mn\" (UniqueName: \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.487130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.487113 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.489273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.489251 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.547472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.547444 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8mn\" (UniqueName: \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\") pod \"maas-api-revoke-sub-65c09441-sjlfk\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:18.637153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:18.637126 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:19.012741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:19.012716 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk"] Nov 03 22:38:19.014356 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:19.014323 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb02a71_e102_46af_80d8_bc02f2e7454a.slice/crio-923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059 WatchSource:0}: Error finding container 923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059: Status 404 returned error can't find the container with id 923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059 Nov 03 22:38:19.154583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:19.154555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" event={"ID":"1eb02a71-e102-46af-80d8-bc02f2e7454a","Type":"ContainerStarted","Data":"923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059"} Nov 03 22:38:20.162644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:20.162613 5051 generic.go:356] "Generic (PLEG): container finished" podID="1eb02a71-e102-46af-80d8-bc02f2e7454a" containerID="adfaa95479b431adccf5002c1badf4e6595a84afbd9505d3231db4e4ff272e3c" exitCode=0 Nov 03 22:38:20.163056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:20.162703 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" event={"ID":"1eb02a71-e102-46af-80d8-bc02f2e7454a","Type":"ContainerDied","Data":"adfaa95479b431adccf5002c1badf4e6595a84afbd9505d3231db4e4ff272e3c"} Nov 03 22:38:21.340360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.340339 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:21.421220 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.421147 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\" (UniqueName: \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\") pod \"1eb02a71-e102-46af-80d8-bc02f2e7454a\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " Nov 03 22:38:21.421380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.421345 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\") pod \"1eb02a71-e102-46af-80d8-bc02f2e7454a\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " Nov 03 22:38:21.421459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.421419 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\") pod \"1eb02a71-e102-46af-80d8-bc02f2e7454a\" (UID: \"1eb02a71-e102-46af-80d8-bc02f2e7454a\") " Nov 03 22:38:21.421748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.421726 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca" pod "1eb02a71-e102-46af-80d8-bc02f2e7454a" (UID: "1eb02a71-e102-46af-80d8-bc02f2e7454a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:21.421829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.421812 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:21.423459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.423442 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn" pod "1eb02a71-e102-46af-80d8-bc02f2e7454a" (UID: "1eb02a71-e102-46af-80d8-bc02f2e7454a"). InnerVolumeSpecName "kube-api-access-7r8mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:21.423510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.423470 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca" pod "1eb02a71-e102-46af-80d8-bc02f2e7454a" (UID: "1eb02a71-e102-46af-80d8-bc02f2e7454a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:21.522924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.522896 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb02a71-e102-46af-80d8-bc02f2e7454a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:21.522924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:21.522922 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8mn\" (UniqueName: \"kubernetes.io/projected/1eb02a71-e102-46af-80d8-bc02f2e7454a-kube-api-access-7r8mn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:21.623795 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:38:21.623755 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:38:22.178777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:22.178745 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" Nov 03 22:38:22.178942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:22.178786 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk" event={"ID":"1eb02a71-e102-46af-80d8-bc02f2e7454a","Type":"ContainerDied","Data":"923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059"} Nov 03 22:38:22.178942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:22.178867 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923d54b26c1e8be7f5d3f889e8c85e673ada08ceb145dc4d62880479e850a059" Nov 03 22:38:29.077793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.077759 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj"] Nov 03 22:38:29.078300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.078287 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1eb02a71-e102-46af-80d8-bc02f2e7454a" containerName="revoke-keys" Nov 03 22:38:29.078344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.078301 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb02a71-e102-46af-80d8-bc02f2e7454a" containerName="revoke-keys" Nov 03 22:38:29.078428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.078419 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb02a71-e102-46af-80d8-bc02f2e7454a" containerName="revoke-keys" Nov 03 22:38:29.081896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.081881 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.092219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.092197 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k"] Nov 03 22:38:29.095686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.095672 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.134718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.134671 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:29.138458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.138400 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:29.143376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.143352 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj"] Nov 03 22:38:29.178450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.178428 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k"] Nov 03 22:38:29.203267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203233 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.203395 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203280 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.203461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203427 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.203521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203471 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xv4\" (UniqueName: \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.203583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203558 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqj4d\" (UniqueName: \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.203633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.203616 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.304618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304592 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.304766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304634 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xv4\" (UniqueName: \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.304766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304657 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqj4d\" (UniqueName: \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.304766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304686 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.304918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304859 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.304918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.304892 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.305414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.305386 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.305528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.305460 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.307418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.307398 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.307778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.307762 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.364812 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.364790 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqj4d\" (UniqueName: \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\") pod \"maas-api-revoke-sub-b5030940-94nmj\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.365047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.365029 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xv4\" (UniqueName: \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\") pod \"maas-api-revoke-sub-59672595-72f6k\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.395561 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.395538 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:29.409557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.408362 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:29.784407 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.784331 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj"] Nov 03 22:38:29.819966 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:29.819930 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1392b7_3a47_41b0_acdc_460b23b24e42.slice/crio-67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06 WatchSource:0}: Error finding container 67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06: Status 404 returned error can't find the container with id 67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06 Nov 03 22:38:29.835266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:29.835237 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k"] Nov 03 22:38:29.836037 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:29.835974 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecff6f2_9ef9_48a4_9b6a_e1726d7f267c.slice/crio-dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f WatchSource:0}: Error finding container dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f: Status 404 returned error can't find the container with id dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f Nov 03 22:38:30.231516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:30.231478 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" event={"ID":"5b1392b7-3a47-41b0-acdc-460b23b24e42","Type":"ContainerStarted","Data":"1536dcc407cbeca0616f928be28a5bbaec606738efc9b8916a108946caa89766"} Nov 03 22:38:30.231973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:30.231580 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" event={"ID":"5b1392b7-3a47-41b0-acdc-460b23b24e42","Type":"ContainerStarted","Data":"67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06"} Nov 03 22:38:30.233000 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:30.232954 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" event={"ID":"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c","Type":"ContainerStarted","Data":"dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f"} Nov 03 22:38:30.348775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:30.348664 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" podStartSLOduration=1.348650386 podStartE2EDuration="1.348650386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:30.345770475 +0000 UTC m=+984.083114700" watchObservedRunningTime="2026-11-03 22:38:30.348650386 +0000 UTC m=+984.085994611" Nov 03 22:38:31.240822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:31.240782 5051 generic.go:356] "Generic (PLEG): container finished" podID="5b1392b7-3a47-41b0-acdc-460b23b24e42" containerID="1536dcc407cbeca0616f928be28a5bbaec606738efc9b8916a108946caa89766" exitCode=0 Nov 03 22:38:31.241368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:31.240869 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" event={"ID":"5b1392b7-3a47-41b0-acdc-460b23b24e42","Type":"ContainerDied","Data":"1536dcc407cbeca0616f928be28a5bbaec606738efc9b8916a108946caa89766"} Nov 03 22:38:31.242775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:31.242749 5051 generic.go:356] "Generic (PLEG): container finished" podID="7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" containerID="871b8ec096d8ae8e97799c5c3585be279aa39c1359df477aa115a0a49bd12132" exitCode=0 Nov 03 22:38:31.242945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:31.242816 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" event={"ID":"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c","Type":"ContainerDied","Data":"871b8ec096d8ae8e97799c5c3585be279aa39c1359df477aa115a0a49bd12132"} Nov 03 22:38:32.453240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.453219 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:32.455797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.455780 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:32.543505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543427 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\" (UniqueName: \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\") pod \"5b1392b7-3a47-41b0-acdc-460b23b24e42\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " Nov 03 22:38:32.543505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543498 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\") pod \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " Nov 03 22:38:32.543726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543526 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\") pod \"5b1392b7-3a47-41b0-acdc-460b23b24e42\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " Nov 03 22:38:32.543726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543627 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\" (UniqueName: \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\") pod \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " Nov 03 22:38:32.543726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543689 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\") pod \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\" (UID: \"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c\") " Nov 03 22:38:32.543949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543761 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\") pod \"5b1392b7-3a47-41b0-acdc-460b23b24e42\" (UID: \"5b1392b7-3a47-41b0-acdc-460b23b24e42\") " Nov 03 22:38:32.544049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.543964 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca" pod "5b1392b7-3a47-41b0-acdc-460b23b24e42" (UID: "5b1392b7-3a47-41b0-acdc-460b23b24e42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:32.544245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.544215 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca" pod "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" (UID: "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:32.544646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.544622 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.544741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.544652 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.545962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.545936 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca" pod "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" (UID: "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:32.546095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.546072 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d" pod "5b1392b7-3a47-41b0-acdc-460b23b24e42" (UID: "5b1392b7-3a47-41b0-acdc-460b23b24e42"). InnerVolumeSpecName "kube-api-access-vqj4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:32.546539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.546516 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca" pod "5b1392b7-3a47-41b0-acdc-460b23b24e42" (UID: "5b1392b7-3a47-41b0-acdc-460b23b24e42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:32.546617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.546585 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4" pod "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" (UID: "7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c"). InnerVolumeSpecName "kube-api-access-s8xv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:32.645594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.645560 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.645594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.645590 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xv4\" (UniqueName: \"kubernetes.io/projected/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c-kube-api-access-s8xv4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.645782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.645606 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1392b7-3a47-41b0-acdc-460b23b24e42-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:32.645782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:32.645618 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqj4d\" (UniqueName: \"kubernetes.io/projected/5b1392b7-3a47-41b0-acdc-460b23b24e42-kube-api-access-vqj4d\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.261191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.261157 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" Nov 03 22:38:33.261401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.261187 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj" event={"ID":"5b1392b7-3a47-41b0-acdc-460b23b24e42","Type":"ContainerDied","Data":"67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06"} Nov 03 22:38:33.261401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.261241 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b19502b942d02c8a9adf76d7946d7debcea8815bd004596e4f001813ed8b06" Nov 03 22:38:33.262821 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.262796 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" Nov 03 22:38:33.262955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.262796 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k" event={"ID":"7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c","Type":"ContainerDied","Data":"dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f"} Nov 03 22:38:33.262955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.262925 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7daad48100d2e42faef33f494dafec715219faa6bbf5b18b346d17abb9fa3f" Nov 03 22:38:33.450435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.450403 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq"] Nov 03 22:38:33.450938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.450924 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5b1392b7-3a47-41b0-acdc-460b23b24e42" containerName="revoke-keys" Nov 03 22:38:33.450984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.450939 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1392b7-3a47-41b0-acdc-460b23b24e42" containerName="revoke-keys" Nov 03 22:38:33.450984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.450959 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" containerName="revoke-keys" Nov 03 22:38:33.450984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.450963 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" containerName="revoke-keys" Nov 03 22:38:33.451112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.451092 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1392b7-3a47-41b0-acdc-460b23b24e42" containerName="revoke-keys" Nov 03 22:38:33.451112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.451100 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" containerName="revoke-keys" Nov 03 22:38:33.454716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.454698 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.507236 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.507206 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:33.507555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.507454 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:33.511612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.511563 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq"] Nov 03 22:38:33.555965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.555939 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzfsf\" (UniqueName: \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.556081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.555976 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.556248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.556225 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.657509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.657480 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.657648 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.657525 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzfsf\" (UniqueName: \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.657648 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.657555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.658205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.658183 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.660073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.660058 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.698258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.694026 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4"] Nov 03 22:38:33.700841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.700819 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.719175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.719151 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzfsf\" (UniqueName: \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\") pod \"maas-api-revoke-sub-1a18197e-zp6bq\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.753617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.753595 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4"] Nov 03 22:38:33.759125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.759097 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.759320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.759300 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.759391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.759368 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.770808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.770760 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:33.860432 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.860407 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.860585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.860566 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.860650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.860628 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.861658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.861635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.863141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.863118 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:33.921305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:33.921277 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\") pod \"maas-api-revoke-sub-bb95a276-lvfl4\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:34.014301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:34.014279 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:34.157748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:34.157719 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq"] Nov 03 22:38:34.159121 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:34.159098 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b8994_0ec6_450a_8f26_250a876eb277.slice/crio-210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191 WatchSource:0}: Error finding container 210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191: Status 404 returned error can't find the container with id 210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191 Nov 03 22:38:34.271624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:34.271587 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" event={"ID":"1b4b8994-0ec6-450a-8f26-250a876eb277","Type":"ContainerStarted","Data":"210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191"} Nov 03 22:38:34.393151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:34.393124 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4"] Nov 03 22:38:34.394955 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:34.394924 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b02f11_c59b_4d1d_bb36_c01594488dff.slice/crio-bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857 WatchSource:0}: Error finding container bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857: Status 404 returned error can't find the container with id bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857 Nov 03 22:38:35.281311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:35.281279 5051 generic.go:356] "Generic (PLEG): container finished" podID="1b4b8994-0ec6-450a-8f26-250a876eb277" containerID="fa1b35cb5a99ca3aa1236f4d4dde4e510c320a5d860d04bbee8d9c5fc37ff74e" exitCode=0 Nov 03 22:38:35.281819 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:35.281371 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" event={"ID":"1b4b8994-0ec6-450a-8f26-250a876eb277","Type":"ContainerDied","Data":"fa1b35cb5a99ca3aa1236f4d4dde4e510c320a5d860d04bbee8d9c5fc37ff74e"} Nov 03 22:38:35.283140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:35.283117 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" event={"ID":"89b02f11-c59b-4d1d-bb36-c01594488dff","Type":"ContainerStarted","Data":"5ed192c0406007a1742ed0f837c6dc5f444822a83d1af54cd2ed66f1fd21f209"} Nov 03 22:38:35.283282 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:35.283155 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" event={"ID":"89b02f11-c59b-4d1d-bb36-c01594488dff","Type":"ContainerStarted","Data":"bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857"} Nov 03 22:38:35.513904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:35.513854 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" podStartSLOduration=2.513841044 podStartE2EDuration="2.513841044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:35.512087887 +0000 UTC m=+989.249432114" watchObservedRunningTime="2026-11-03 22:38:35.513841044 +0000 UTC m=+989.251185268" Nov 03 22:38:36.293451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.293418 5051 generic.go:356] "Generic (PLEG): container finished" podID="89b02f11-c59b-4d1d-bb36-c01594488dff" containerID="5ed192c0406007a1742ed0f837c6dc5f444822a83d1af54cd2ed66f1fd21f209" exitCode=0 Nov 03 22:38:36.293894 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.293510 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" event={"ID":"89b02f11-c59b-4d1d-bb36-c01594488dff","Type":"ContainerDied","Data":"5ed192c0406007a1742ed0f837c6dc5f444822a83d1af54cd2ed66f1fd21f209"} Nov 03 22:38:36.470701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.470672 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:36.593340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.593247 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\") pod \"1b4b8994-0ec6-450a-8f26-250a876eb277\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " Nov 03 22:38:36.593524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.593346 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\" (UniqueName: \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\") pod \"1b4b8994-0ec6-450a-8f26-250a876eb277\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " Nov 03 22:38:36.593524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.593416 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\") pod \"1b4b8994-0ec6-450a-8f26-250a876eb277\" (UID: \"1b4b8994-0ec6-450a-8f26-250a876eb277\") " Nov 03 22:38:36.593870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.593842 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca" pod "1b4b8994-0ec6-450a-8f26-250a876eb277" (UID: "1b4b8994-0ec6-450a-8f26-250a876eb277"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:36.595613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.595578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca" pod "1b4b8994-0ec6-450a-8f26-250a876eb277" (UID: "1b4b8994-0ec6-450a-8f26-250a876eb277"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:36.595776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.595759 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf" pod "1b4b8994-0ec6-450a-8f26-250a876eb277" (UID: "1b4b8994-0ec6-450a-8f26-250a876eb277"). InnerVolumeSpecName "kube-api-access-dzfsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:36.695252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.695223 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:36.695252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.695250 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4b8994-0ec6-450a-8f26-250a876eb277-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:36.695411 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:36.695267 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzfsf\" (UniqueName: \"kubernetes.io/projected/1b4b8994-0ec6-450a-8f26-250a876eb277-kube-api-access-dzfsf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:37.302849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.302819 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" Nov 03 22:38:37.303331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.302880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq" event={"ID":"1b4b8994-0ec6-450a-8f26-250a876eb277","Type":"ContainerDied","Data":"210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191"} Nov 03 22:38:37.303331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.302933 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210156f75886104ab931f94afa28b9c6529fbcd38d64c1ada45e06effe06b191" Nov 03 22:38:37.454389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.454361 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:37.605398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.605362 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\") pod \"89b02f11-c59b-4d1d-bb36-c01594488dff\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " Nov 03 22:38:37.605579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.605521 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\") pod \"89b02f11-c59b-4d1d-bb36-c01594488dff\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " Nov 03 22:38:37.605678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.605626 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\") pod \"89b02f11-c59b-4d1d-bb36-c01594488dff\" (UID: \"89b02f11-c59b-4d1d-bb36-c01594488dff\") " Nov 03 22:38:37.606169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.606086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca" pod "89b02f11-c59b-4d1d-bb36-c01594488dff" (UID: "89b02f11-c59b-4d1d-bb36-c01594488dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:37.608026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.607965 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8" pod "89b02f11-c59b-4d1d-bb36-c01594488dff" (UID: "89b02f11-c59b-4d1d-bb36-c01594488dff"). InnerVolumeSpecName "kube-api-access-bsvj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:37.608769 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.608744 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca" pod "89b02f11-c59b-4d1d-bb36-c01594488dff" (UID: "89b02f11-c59b-4d1d-bb36-c01594488dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:37.707214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.707186 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsvj8\" (UniqueName: \"kubernetes.io/projected/89b02f11-c59b-4d1d-bb36-c01594488dff-kube-api-access-bsvj8\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:37.707214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.707213 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:37.707439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:37.707224 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b02f11-c59b-4d1d-bb36-c01594488dff-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:38.316606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:38.316569 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" event={"ID":"89b02f11-c59b-4d1d-bb36-c01594488dff","Type":"ContainerDied","Data":"bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857"} Nov 03 22:38:38.317181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:38.317159 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf047880901a3463802167f6c66e0a96c7c8830396bcd4927b515c34093a3857" Nov 03 22:38:38.317366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:38.317342 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4" Nov 03 22:38:46.361230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361196 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh"] Nov 03 22:38:46.361755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361739 5051 cpu_manager.go:414] "containerMap: removing container" podUID="89b02f11-c59b-4d1d-bb36-c01594488dff" containerName="revoke-keys" Nov 03 22:38:46.361755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361755 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b02f11-c59b-4d1d-bb36-c01594488dff" containerName="revoke-keys" Nov 03 22:38:46.361884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361777 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1b4b8994-0ec6-450a-8f26-250a876eb277" containerName="revoke-keys" Nov 03 22:38:46.361884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361783 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4b8994-0ec6-450a-8f26-250a876eb277" containerName="revoke-keys" Nov 03 22:38:46.361884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361875 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4b8994-0ec6-450a-8f26-250a876eb277" containerName="revoke-keys" Nov 03 22:38:46.362070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.361889 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b02f11-c59b-4d1d-bb36-c01594488dff" containerName="revoke-keys" Nov 03 22:38:46.365237 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.365219 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.423280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.423253 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh"] Nov 03 22:38:46.433217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.433193 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnlv\" (UniqueName: \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.433330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.433257 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.433369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.433345 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.468047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.468025 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:46.468047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.468026 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:46.534195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.534169 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnlv\" (UniqueName: \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.534359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.534219 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.534359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.534269 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.534920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.534896 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.536825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.536795 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.595339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.595306 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnlv\" (UniqueName: \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\") pod \"maas-api-revoke-sub-a72a4bc1-gxswh\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:46.681509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:46.681446 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:47.062908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:47.062878 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh"] Nov 03 22:38:47.063281 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:47.063258 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff6e5bc_d0ad_4e71_a75b_0946ccef055c.slice/crio-ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7 WatchSource:0}: Error finding container ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7: Status 404 returned error can't find the container with id ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7 Nov 03 22:38:47.393467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:47.393412 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" event={"ID":"bff6e5bc-d0ad-4e71-a75b-0946ccef055c","Type":"ContainerStarted","Data":"60ca547adca6ffa38110f5ec148769193392377a1741d5cc4426c5dbe670effd"} Nov 03 22:38:47.393908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:47.393494 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" event={"ID":"bff6e5bc-d0ad-4e71-a75b-0946ccef055c","Type":"ContainerStarted","Data":"ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7"} Nov 03 22:38:47.510924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:47.510869 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" podStartSLOduration=1.510852313 podStartE2EDuration="1.510852313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:47.508068291 +0000 UTC m=+1001.245412518" watchObservedRunningTime="2026-11-03 22:38:47.510852313 +0000 UTC m=+1001.248196539" Nov 03 22:38:48.412823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.412783 5051 generic.go:356] "Generic (PLEG): container finished" podID="bff6e5bc-d0ad-4e71-a75b-0946ccef055c" containerID="60ca547adca6ffa38110f5ec148769193392377a1741d5cc4426c5dbe670effd" exitCode=0 Nov 03 22:38:48.415469 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.415441 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66"] Nov 03 22:38:48.419329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.419307 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" event={"ID":"bff6e5bc-d0ad-4e71-a75b-0946ccef055c","Type":"ContainerDied","Data":"60ca547adca6ffa38110f5ec148769193392377a1741d5cc4426c5dbe670effd"} Nov 03 22:38:48.419416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.419388 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.452238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.452208 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.452434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.452258 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.452434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.452424 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9kr\" (UniqueName: \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.474066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.473859 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66"] Nov 03 22:38:48.553117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.553086 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9kr\" (UniqueName: \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.553279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.553168 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.553279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.553204 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.553926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.553903 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.555846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.555821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.613885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.613857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9kr\" (UniqueName: \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\") pod \"maas-api-revoke-sub-49c57c21-bjx66\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:48.733568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:48.733483 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:49.113274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.113247 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66"] Nov 03 22:38:49.114540 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:49.114517 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5946c2b6_c802_41b8_b0ea_2f0edc59a94a.slice/crio-36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e WatchSource:0}: Error finding container 36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e: Status 404 returned error can't find the container with id 36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e Nov 03 22:38:49.421875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.421841 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" event={"ID":"5946c2b6-c802-41b8-b0ea-2f0edc59a94a","Type":"ContainerStarted","Data":"cac533a2e02fc28e27c78140f4214317ed72d6ad6ea023fc37ed5a141ba520a3"} Nov 03 22:38:49.422379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.421895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" event={"ID":"5946c2b6-c802-41b8-b0ea-2f0edc59a94a","Type":"ContainerStarted","Data":"36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e"} Nov 03 22:38:49.541277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.541171 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" podStartSLOduration=1.5411505170000002 podStartE2EDuration="1.541150517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:49.536682902 +0000 UTC m=+1003.274027140" watchObservedRunningTime="2026-11-03 22:38:49.541150517 +0000 UTC m=+1003.278494742" Nov 03 22:38:49.556027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.555981 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:49.563277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.563245 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\" (UniqueName: \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\") pod \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " Nov 03 22:38:49.563427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.563381 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\") pod \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " Nov 03 22:38:49.563536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.563520 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\") pod \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\" (UID: \"bff6e5bc-d0ad-4e71-a75b-0946ccef055c\") " Nov 03 22:38:49.563755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.563729 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca" pod "bff6e5bc-d0ad-4e71-a75b-0946ccef055c" (UID: "bff6e5bc-d0ad-4e71-a75b-0946ccef055c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:49.563943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.563921 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:49.565458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.565427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv" pod "bff6e5bc-d0ad-4e71-a75b-0946ccef055c" (UID: "bff6e5bc-d0ad-4e71-a75b-0946ccef055c"). InnerVolumeSpecName "kube-api-access-fhnlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:49.566046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.566028 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca" pod "bff6e5bc-d0ad-4e71-a75b-0946ccef055c" (UID: "bff6e5bc-d0ad-4e71-a75b-0946ccef055c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:49.665196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.665099 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnlv\" (UniqueName: \"kubernetes.io/projected/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-kube-api-access-fhnlv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:49.665196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:49.665137 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff6e5bc-d0ad-4e71-a75b-0946ccef055c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:50.429419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:50.429389 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" Nov 03 22:38:50.429871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:50.429420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh" event={"ID":"bff6e5bc-d0ad-4e71-a75b-0946ccef055c","Type":"ContainerDied","Data":"ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7"} Nov 03 22:38:50.429871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:50.429486 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc71b71bb208b89ab341ffbc3550c80f576c1bc3f258a3555998f84b62c57c7" Nov 03 22:38:50.431311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:50.431287 5051 generic.go:356] "Generic (PLEG): container finished" podID="5946c2b6-c802-41b8-b0ea-2f0edc59a94a" containerID="cac533a2e02fc28e27c78140f4214317ed72d6ad6ea023fc37ed5a141ba520a3" exitCode=0 Nov 03 22:38:50.431427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:50.431373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" event={"ID":"5946c2b6-c802-41b8-b0ea-2f0edc59a94a","Type":"ContainerDied","Data":"cac533a2e02fc28e27c78140f4214317ed72d6ad6ea023fc37ed5a141ba520a3"} Nov 03 22:38:51.376796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.376763 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs"] Nov 03 22:38:51.377375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.377360 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bff6e5bc-d0ad-4e71-a75b-0946ccef055c" containerName="revoke-keys" Nov 03 22:38:51.377421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.377377 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff6e5bc-d0ad-4e71-a75b-0946ccef055c" containerName="revoke-keys" Nov 03 22:38:51.377476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.377467 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff6e5bc-d0ad-4e71-a75b-0946ccef055c" containerName="revoke-keys" Nov 03 22:38:51.380929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.380912 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.484705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.484664 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs"] Nov 03 22:38:51.587408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.587368 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.587602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.587457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.587669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.587648 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wrb\" (UniqueName: \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.625117 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:38:51.625078 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:38:51.625614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.625591 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:51.688913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.688826 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\" (UniqueName: \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\") pod \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " Nov 03 22:38:51.689124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.688938 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\") pod \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " Nov 03 22:38:51.689318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.689293 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.689401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.689381 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.689600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.689546 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wrb\" (UniqueName: \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.690176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.690152 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.691379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.691346 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr" pod "5946c2b6-c802-41b8-b0ea-2f0edc59a94a" (UID: "5946c2b6-c802-41b8-b0ea-2f0edc59a94a"). InnerVolumeSpecName "kube-api-access-kb9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:51.691630 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.691595 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca" pod "5946c2b6-c802-41b8-b0ea-2f0edc59a94a" (UID: "5946c2b6-c802-41b8-b0ea-2f0edc59a94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:51.692610 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.692587 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.753214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.753186 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wrb\" (UniqueName: \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\") pod \"maas-api-revoke-sub-9590ba86-c28qs\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:51.790287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.790252 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\") pod \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\" (UID: \"5946c2b6-c802-41b8-b0ea-2f0edc59a94a\") " Nov 03 22:38:51.790624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.790601 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca" pod "5946c2b6-c802-41b8-b0ea-2f0edc59a94a" (UID: "5946c2b6-c802-41b8-b0ea-2f0edc59a94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:51.790813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.790796 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb9kr\" (UniqueName: \"kubernetes.io/projected/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-kube-api-access-kb9kr\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:51.790847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.790821 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:51.790847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.790836 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5946c2b6-c802-41b8-b0ea-2f0edc59a94a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:51.996978 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:51.996893 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:52.376352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:52.376325 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs"] Nov 03 22:38:52.377599 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:38:52.377574 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38231b2e_9b93_4e6e_9cc3_5130fec353e2.slice/crio-59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484 WatchSource:0}: Error finding container 59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484: Status 404 returned error can't find the container with id 59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484 Nov 03 22:38:52.446650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:52.446625 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" Nov 03 22:38:52.446798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:52.446659 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66" event={"ID":"5946c2b6-c802-41b8-b0ea-2f0edc59a94a","Type":"ContainerDied","Data":"36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e"} Nov 03 22:38:52.446798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:52.446710 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36d0e3a68282fb189f87604bda0136abf9d3036ea2ef60719c2f36bd847cd23e" Nov 03 22:38:52.448172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:52.448149 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" event={"ID":"38231b2e-9b93-4e6e-9cc3-5130fec353e2","Type":"ContainerStarted","Data":"59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484"} Nov 03 22:38:53.456620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:53.456583 5051 generic.go:356] "Generic (PLEG): container finished" podID="38231b2e-9b93-4e6e-9cc3-5130fec353e2" containerID="fbf24e005fae124285e9405355b045b95ec1e74856c884a79c7114d63062b194" exitCode=0 Nov 03 22:38:53.457055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:53.456668 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" event={"ID":"38231b2e-9b93-4e6e-9cc3-5130fec353e2","Type":"ContainerDied","Data":"fbf24e005fae124285e9405355b045b95ec1e74856c884a79c7114d63062b194"} Nov 03 22:38:54.619101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.619071 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:54.716537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.716451 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\") pod \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " Nov 03 22:38:54.716537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.716529 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\" (UniqueName: \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\") pod \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " Nov 03 22:38:54.716767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.716570 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\") pod \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\" (UID: \"38231b2e-9b93-4e6e-9cc3-5130fec353e2\") " Nov 03 22:38:54.716948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.716920 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca" pod "38231b2e-9b93-4e6e-9cc3-5130fec353e2" (UID: "38231b2e-9b93-4e6e-9cc3-5130fec353e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:54.717155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.717134 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:54.718903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.718870 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca" pod "38231b2e-9b93-4e6e-9cc3-5130fec353e2" (UID: "38231b2e-9b93-4e6e-9cc3-5130fec353e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:54.719046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.718909 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb" pod "38231b2e-9b93-4e6e-9cc3-5130fec353e2" (UID: "38231b2e-9b93-4e6e-9cc3-5130fec353e2"). InnerVolumeSpecName "kube-api-access-g9wrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:54.818327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.818294 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wrb\" (UniqueName: \"kubernetes.io/projected/38231b2e-9b93-4e6e-9cc3-5130fec353e2-kube-api-access-g9wrb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:54.818327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:54.818322 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38231b2e-9b93-4e6e-9cc3-5130fec353e2-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:38:55.472575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:55.472537 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" event={"ID":"38231b2e-9b93-4e6e-9cc3-5130fec353e2","Type":"ContainerDied","Data":"59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484"} Nov 03 22:38:55.472736 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:55.472591 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ec05718a73a8e71bc6c637033bd666175b06d6747a48725ce1fd913279c484" Nov 03 22:38:55.472736 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:55.472555 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs" Nov 03 22:38:59.250925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.250890 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc"] Nov 03 22:38:59.251430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251418 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5946c2b6-c802-41b8-b0ea-2f0edc59a94a" containerName="revoke-keys" Nov 03 22:38:59.251470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251431 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5946c2b6-c802-41b8-b0ea-2f0edc59a94a" containerName="revoke-keys" Nov 03 22:38:59.251516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251470 5051 cpu_manager.go:414] "containerMap: removing container" podUID="38231b2e-9b93-4e6e-9cc3-5130fec353e2" containerName="revoke-keys" Nov 03 22:38:59.251516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251475 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38231b2e-9b93-4e6e-9cc3-5130fec353e2" containerName="revoke-keys" Nov 03 22:38:59.251582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251572 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5946c2b6-c802-41b8-b0ea-2f0edc59a94a" containerName="revoke-keys" Nov 03 22:38:59.251615 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.251588 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="38231b2e-9b93-4e6e-9cc3-5130fec353e2" containerName="revoke-keys" Nov 03 22:38:59.255164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.255146 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.308029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.307979 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:59.357677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.357646 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:38:59.361513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.361489 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc"] Nov 03 22:38:59.365345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.365318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.365452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.365434 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47njl\" (UniqueName: \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.365527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.365510 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.466363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.466333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47njl\" (UniqueName: \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.466363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.466368 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.466732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.466699 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.467494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.467469 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.473455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.470379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.529961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.529928 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47njl\" (UniqueName: \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\") pod \"maas-api-revoke-sub-603aa8b5-q42mc\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.546474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.546444 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx"] Nov 03 22:38:59.550494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.550480 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.568940 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.568911 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.569085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.568968 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76kt\" (UniqueName: \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.569172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.569155 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.575681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.575662 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:38:59.674255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.674154 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.675336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.675193 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.675798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.675779 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76kt\" (UniqueName: \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.676056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.676038 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.677304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.677285 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx"] Nov 03 22:38:59.702458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.702414 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.737559 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.737492 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76kt\" (UniqueName: \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\") pod \"maas-api-revoke-sub-84f7aa5b-gs4hx\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:38:59.864683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:38:59.864380 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:39:00.015793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.015763 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc"] Nov 03 22:39:00.314168 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.310417 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx"] Nov 03 22:39:00.556087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.555961 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" event={"ID":"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3","Type":"ContainerStarted","Data":"ad6d22dcaee7c0ebcc57312d52f74aba0f0a1a34754bf6e4c2a72fc49a1f388b"} Nov 03 22:39:00.556087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.556052 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" event={"ID":"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3","Type":"ContainerStarted","Data":"91a5d00ba8461cfa663b83f41cc9c523126233f8702bbdc97c2462919c0a00dd"} Nov 03 22:39:00.572215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.558857 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" event={"ID":"a11dedf8-dbca-4fca-949f-f9eae9007cee","Type":"ContainerStarted","Data":"979c594ea406fc7865b7085528e4aa978517069f79769b5dba54e0c7593ba27e"} Nov 03 22:39:00.699015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:00.693295 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" podStartSLOduration=1.69327479 podStartE2EDuration="1.69327479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:00.674576198 +0000 UTC m=+1014.411920424" watchObservedRunningTime="2026-11-03 22:39:00.69327479 +0000 UTC m=+1014.430619016" Nov 03 22:39:01.566787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:01.566753 5051 generic.go:356] "Generic (PLEG): container finished" podID="a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" containerID="ad6d22dcaee7c0ebcc57312d52f74aba0f0a1a34754bf6e4c2a72fc49a1f388b" exitCode=0 Nov 03 22:39:01.567257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:01.566809 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" event={"ID":"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3","Type":"ContainerDied","Data":"ad6d22dcaee7c0ebcc57312d52f74aba0f0a1a34754bf6e4c2a72fc49a1f388b"} Nov 03 22:39:01.568371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:01.568349 5051 generic.go:356] "Generic (PLEG): container finished" podID="a11dedf8-dbca-4fca-949f-f9eae9007cee" containerID="4864b0f474dd4804617fec5018de53e1bc1c78b5a325ab6abcc1906a1c6c151e" exitCode=0 Nov 03 22:39:01.568444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:01.568389 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" event={"ID":"a11dedf8-dbca-4fca-949f-f9eae9007cee","Type":"ContainerDied","Data":"4864b0f474dd4804617fec5018de53e1bc1c78b5a325ab6abcc1906a1c6c151e"} Nov 03 22:39:02.729035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.729010 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:39:02.769683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.769659 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:39:02.855204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855175 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\") pod \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " Nov 03 22:39:02.855386 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\" (UniqueName: \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\") pod \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " Nov 03 22:39:02.855386 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855362 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\") pod \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\" (UID: \"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3\") " Nov 03 22:39:02.855502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855471 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\") pod \"a11dedf8-dbca-4fca-949f-f9eae9007cee\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " Nov 03 22:39:02.855579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855562 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\") pod \"a11dedf8-dbca-4fca-949f-f9eae9007cee\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " Nov 03 22:39:02.855635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.855606 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\" (UniqueName: \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\") pod \"a11dedf8-dbca-4fca-949f-f9eae9007cee\" (UID: \"a11dedf8-dbca-4fca-949f-f9eae9007cee\") " Nov 03 22:39:02.856232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.856046 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca" pod "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" (UID: "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:02.856232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.856180 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca" pod "a11dedf8-dbca-4fca-949f-f9eae9007cee" (UID: "a11dedf8-dbca-4fca-949f-f9eae9007cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:02.856875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.856841 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:02.856875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.856869 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:02.857962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.857927 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl" pod "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" (UID: "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3"). InnerVolumeSpecName "kube-api-access-47njl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:02.858230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.858206 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca" pod "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" (UID: "a60dfae6-adb8-4e77-8c18-6aa9a4be00a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:02.858319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.858280 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt" pod "a11dedf8-dbca-4fca-949f-f9eae9007cee" (UID: "a11dedf8-dbca-4fca-949f-f9eae9007cee"). InnerVolumeSpecName "kube-api-access-j76kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:02.863155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.863132 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca" pod "a11dedf8-dbca-4fca-949f-f9eae9007cee" (UID: "a11dedf8-dbca-4fca-949f-f9eae9007cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:02.957602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.957572 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11dedf8-dbca-4fca-949f-f9eae9007cee-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:02.957602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.957600 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76kt\" (UniqueName: \"kubernetes.io/projected/a11dedf8-dbca-4fca-949f-f9eae9007cee-kube-api-access-j76kt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:02.957602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.957611 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:02.957602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:02.957619 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47njl\" (UniqueName: \"kubernetes.io/projected/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3-kube-api-access-47njl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:03.584702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.584672 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" Nov 03 22:39:03.584903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.584681 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc" event={"ID":"a60dfae6-adb8-4e77-8c18-6aa9a4be00a3","Type":"ContainerDied","Data":"91a5d00ba8461cfa663b83f41cc9c523126233f8702bbdc97c2462919c0a00dd"} Nov 03 22:39:03.584903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.584789 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a5d00ba8461cfa663b83f41cc9c523126233f8702bbdc97c2462919c0a00dd" Nov 03 22:39:03.586187 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.586165 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" Nov 03 22:39:03.586187 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.586180 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx" event={"ID":"a11dedf8-dbca-4fca-949f-f9eae9007cee","Type":"ContainerDied","Data":"979c594ea406fc7865b7085528e4aa978517069f79769b5dba54e0c7593ba27e"} Nov 03 22:39:03.586340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:03.586224 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979c594ea406fc7865b7085528e4aa978517069f79769b5dba54e0c7593ba27e" Nov 03 22:39:06.118920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.118876 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss"] Nov 03 22:39:06.119447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.119433 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" containerName="revoke-keys" Nov 03 22:39:06.119491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.119448 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" containerName="revoke-keys" Nov 03 22:39:06.119491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.119468 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a11dedf8-dbca-4fca-949f-f9eae9007cee" containerName="revoke-keys" Nov 03 22:39:06.119491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.119473 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11dedf8-dbca-4fca-949f-f9eae9007cee" containerName="revoke-keys" Nov 03 22:39:06.120230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.120205 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" containerName="revoke-keys" Nov 03 22:39:06.120335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.120239 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11dedf8-dbca-4fca-949f-f9eae9007cee" containerName="revoke-keys" Nov 03 22:39:06.126442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.126414 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.168408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.168382 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:39:06.172323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.172304 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:39:06.172503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.172447 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.175812 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.175793 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.193251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193228 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193258 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jttx\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-kube-api-access-2jttx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193286 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193460 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193499 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c799271b-6fb6-4e67-9e4a-45f296bf84db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193597 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193675 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193630 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193675 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193650 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.193782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.193750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.214732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.214704 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89xv7\"" Nov 03 22:39:06.224021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.223975 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss"] Nov 03 22:39:06.292887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.292854 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:39:06.294786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.294759 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.294786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.294798 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.294973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.294925 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.294971 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c799271b-6fb6-4e67-9e4a-45f296bf84db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295021 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295081 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqfl\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295113 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295162 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295186 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295256 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295281 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295315 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295351 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295385 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295419 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295453 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295493 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295592 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295561 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.295638 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295605 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295638 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295631 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jttx\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-kube-api-access-2jttx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.295702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295664 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295697 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295722 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2w4\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.295809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.295750 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.296674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.296645 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.296780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.296691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.297011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.296922 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.297544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.297520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.297634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.297551 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c799271b-6fb6-4e67-9e4a-45f296bf84db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.297755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.297737 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:39:06.299760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.299731 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.300765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.300740 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.357371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.357345 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.357678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.357655 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jttx\" (UniqueName: \"kubernetes.io/projected/c799271b-6fb6-4e67-9e4a-45f296bf84db-kube-api-access-2jttx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4bjss\" (UID: \"c799271b-6fb6-4e67-9e4a-45f296bf84db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.397245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqfl\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397209 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397234 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397264 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397317 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397353 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397381 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397553 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397602 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397632 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397659 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2w4\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397724 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397763 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.397917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.398286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.398122 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.398403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.398379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.399768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.399733 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.400629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.400604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.400801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.400775 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.400866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.397657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.401101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.401077 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.401200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.401127 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.401806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.401315 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.401806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.401577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.401806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.401756 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.402621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.402201 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.404263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.404242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.405269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.405234 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.458277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.458252 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqfl\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.461020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.460960 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2jqg5\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.461123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.461096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.461167 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.461127 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2w4\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hn9sv\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.492244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.492219 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89xv7\"" Nov 03 22:39:06.492798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.492777 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:06.497168 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.497146 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:06.501145 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.501125 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:06.848984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.848902 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 22:39:06.854960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.854935 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.904412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.904381 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.904585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.904435 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.904882 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.904856 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.905083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.905064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hfr\" (UniqueName: \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.905142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.905110 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.905203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.905187 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:06.907850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.907823 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b64e2e\"/\"distinct-w0-b64e2e-kserve-self-signed-certs\"" Nov 03 22:39:06.911982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.911954 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 22:39:06.957300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.957268 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:39:06.958425 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:06.958265 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c6b8c6_7653_475a_817a_1fd48b6013af.slice/crio-56358af86d1e7f097797ecf9bd5ae81a7e8cea5fcdc322576116356c833e53fd WatchSource:0}: Error finding container 56358af86d1e7f097797ecf9bd5ae81a7e8cea5fcdc322576116356c833e53fd: Status 404 returned error can't find the container with id 56358af86d1e7f097797ecf9bd5ae81a7e8cea5fcdc322576116356c833e53fd Nov 03 22:39:06.961033 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:06.960967 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc799271b_6fb6_4e67_9e4a_45f296bf84db.slice/crio-a03c45be243a1936f29f136f111b372ca1719206f827ae2dcce95aa39b35e2ea WatchSource:0}: Error finding container a03c45be243a1936f29f136f111b372ca1719206f827ae2dcce95aa39b35e2ea: Status 404 returned error can't find the container with id a03c45be243a1936f29f136f111b372ca1719206f827ae2dcce95aa39b35e2ea Nov 03 22:39:06.961841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.961787 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.961966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.961877 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.961966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.961921 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.962151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.962111 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss"] Nov 03 22:39:06.963929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.963900 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.964030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.963960 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.964030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.964007 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.977709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:06.977674 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.009838 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.009904 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.010075 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.010144 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hfr\" (UniqueName: \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.010180 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.010376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.010217 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.011101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.010663 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.015975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.011361 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.015975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.011580 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.015975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.015343 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.022955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.019766 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.026329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.026286 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:07.026428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.026375 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:07.026428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.026422 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:07.073432 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.073293 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hfr\" (UniqueName: \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\") pod \"distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.169478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.169271 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:07.624835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.624739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" event={"ID":"c799271b-6fb6-4e67-9e4a-45f296bf84db","Type":"ContainerStarted","Data":"6f819504a6470904cc5f3c15d0e5406acc3f3252d436665f85a48dc61435cae5"} Nov 03 22:39:07.624835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.624837 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" event={"ID":"c799271b-6fb6-4e67-9e4a-45f296bf84db","Type":"ContainerStarted","Data":"a03c45be243a1936f29f136f111b372ca1719206f827ae2dcce95aa39b35e2ea"} Nov 03 22:39:07.627053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.627024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" event={"ID":"88b4ba0b-c12e-4ae9-9baa-b801e810544f","Type":"ContainerStarted","Data":"90fd2b46f9f08d45c733d59b8f503ac25df2c56b99c0ecf8d6b5a4e995589baf"} Nov 03 22:39:07.627187 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.627081 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" event={"ID":"88b4ba0b-c12e-4ae9-9baa-b801e810544f","Type":"ContainerStarted","Data":"9372deef244010f575ec8d247a471b2f134cdf1388b05dec08ad9fb6b0a23f76"} Nov 03 22:39:07.628931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.628907 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" event={"ID":"90c6b8c6-7653-475a-817a-1fd48b6013af","Type":"ContainerStarted","Data":"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c"} Nov 03 22:39:07.629050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.628947 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" event={"ID":"90c6b8c6-7653-475a-817a-1fd48b6013af","Type":"ContainerStarted","Data":"56358af86d1e7f097797ecf9bd5ae81a7e8cea5fcdc322576116356c833e53fd"} Nov 03 22:39:07.636040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.636014 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 22:39:07.647103 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:07.647050 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94c7597d_73bd_4bdf_8726_f464f6cd51b0.slice/crio-aa60d8d2d792570e601b15c2d95dab78bdf288b48143baca117de8c5f7b6767a WatchSource:0}: Error finding container aa60d8d2d792570e601b15c2d95dab78bdf288b48143baca117de8c5f7b6767a: Status 404 returned error can't find the container with id aa60d8d2d792570e601b15c2d95dab78bdf288b48143baca117de8c5f7b6767a Nov 03 22:39:07.747002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.746762 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podStartSLOduration=1.746746623 podStartE2EDuration="1.746746623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:07.744981114 +0000 UTC m=+1021.482325340" watchObservedRunningTime="2026-11-03 22:39:07.746746623 +0000 UTC m=+1021.484090847" Nov 03 22:39:07.828032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.825677 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk"] Nov 03 22:39:07.865026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.857357 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:07.922447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.921284 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:39:07.922447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:07.921423 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:39:08.050029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.048626 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxtp\" (UniqueName: \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.050029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.048789 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.050029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.048830 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.055065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.050412 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk"] Nov 03 22:39:08.076277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.055880 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podStartSLOduration=2.055863097 podStartE2EDuration="2.055863097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:07.878159734 +0000 UTC m=+1021.615503960" watchObservedRunningTime="2026-11-03 22:39:08.055863097 +0000 UTC m=+1021.793207331" Nov 03 22:39:08.149844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.149753 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxtp\" (UniqueName: \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.150206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.150183 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.150371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.150353 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.171016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.151210 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.171016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.154658 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.252360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.252322 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxtp\" (UniqueName: \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\") pod \"maas-api-revoke-sub-412f84f8-z5xlk\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.333181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.331648 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podStartSLOduration=2.331633426 podStartE2EDuration="2.331633426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:08.143444537 +0000 UTC m=+1021.880788788" watchObservedRunningTime="2026-11-03 22:39:08.331633426 +0000 UTC m=+1022.068977650" Nov 03 22:39:08.519623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.519530 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:08.526013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.520475 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:08.526013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.520535 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:08.526013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.521047 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529125 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529182 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529494 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529531 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529597 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:08.540013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.529615 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:08.748706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.748669 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" event={"ID":"94c7597d-73bd-4bdf-8726-f464f6cd51b0","Type":"ContainerStarted","Data":"6bb79062a8f6415256575f13b1657b72d232da8146d6223b69ce10a2b61a5e97"} Nov 03 22:39:08.749446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.749407 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" event={"ID":"94c7597d-73bd-4bdf-8726-f464f6cd51b0","Type":"ContainerStarted","Data":"aa60d8d2d792570e601b15c2d95dab78bdf288b48143baca117de8c5f7b6767a"} Nov 03 22:39:08.925017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:08.901584 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podStartSLOduration=2.901564401 podStartE2EDuration="2.901564401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:08.900105273 +0000 UTC m=+1022.637449499" watchObservedRunningTime="2026-11-03 22:39:08.901564401 +0000 UTC m=+1022.638908630" Nov 03 22:39:09.170129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.170100 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:09.447578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.447052 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk"] Nov 03 22:39:09.447578 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:09.447451 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d591b68_3bbc_4432_9bf2_57a089da8872.slice/crio-fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0 WatchSource:0}: Error finding container fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0: Status 404 returned error can't find the container with id fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0 Nov 03 22:39:09.521015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.520113 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:09.521015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.520178 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:09.521015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.520612 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:09.521015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.520651 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:09.526015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.521467 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:09.526015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.521509 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:09.784070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:09.783126 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" event={"ID":"2d591b68-3bbc-4432-9bf2-57a089da8872","Type":"ContainerStarted","Data":"fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0"} Nov 03 22:39:10.505078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.505038 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:10.506289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.506258 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:10.506465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.506151 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:10.506578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.506560 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:10.506683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.506194 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:10.506873 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.506852 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:10.837341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.835977 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" event={"ID":"2d591b68-3bbc-4432-9bf2-57a089da8872","Type":"ContainerStarted","Data":"03911032db8f9ba371a635082cd54c17e7b1138701368632ebf7516d4dbf193d"} Nov 03 22:39:11.007645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:10.986447 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" podStartSLOduration=3.9864250549999998 podStartE2EDuration="3.986425055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:10.960402624 +0000 UTC m=+1024.697746850" watchObservedRunningTime="2026-11-03 22:39:10.986425055 +0000 UTC m=+1024.723769280" Nov 03 22:39:11.502016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.493708 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:11.502016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.493784 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:11.513019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.505326 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:11.513019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.505380 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:11.513019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.505656 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:11.513019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.505741 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:11.844206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.844118 5051 generic.go:356] "Generic (PLEG): container finished" podID="2d591b68-3bbc-4432-9bf2-57a089da8872" containerID="03911032db8f9ba371a635082cd54c17e7b1138701368632ebf7516d4dbf193d" exitCode=0 Nov 03 22:39:11.844206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:11.844159 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" event={"ID":"2d591b68-3bbc-4432-9bf2-57a089da8872","Type":"ContainerDied","Data":"03911032db8f9ba371a635082cd54c17e7b1138701368632ebf7516d4dbf193d"} Nov 03 22:39:12.481212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.481175 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn"] Nov 03 22:39:12.486742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.486710 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.496017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.495029 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:12.496017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.495082 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:12.500360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.499496 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:12.500360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.499544 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:12.501960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.501741 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:12.501960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.501782 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:12.550679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.546633 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn"] Nov 03 22:39:12.666024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.663770 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.666024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.663978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894rh\" (UniqueName: \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.666024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.664143 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.768945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.768835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894rh\" (UniqueName: \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.769284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.769260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.769483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.769465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.787049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.779144 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.821522 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.821449 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:12.843038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:12.842941 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894rh\" (UniqueName: \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\") pod \"maas-api-revoke-sub-da02af2f-qwwvn\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:13.155918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.155175 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:13.502015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.498872 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:13.502015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.498935 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:13.513013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.502820 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:13.513013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.502880 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:13.513013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.503634 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:13.513013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.503688 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:13.520017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.516938 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:13.646231 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.646201 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\") pod \"2d591b68-3bbc-4432-9bf2-57a089da8872\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " Nov 03 22:39:13.646644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.646429 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\" (UniqueName: \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\") pod \"2d591b68-3bbc-4432-9bf2-57a089da8872\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " Nov 03 22:39:13.646644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.646482 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\") pod \"2d591b68-3bbc-4432-9bf2-57a089da8872\" (UID: \"2d591b68-3bbc-4432-9bf2-57a089da8872\") " Nov 03 22:39:13.647012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.646974 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca" pod "2d591b68-3bbc-4432-9bf2-57a089da8872" (UID: "2d591b68-3bbc-4432-9bf2-57a089da8872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:13.649130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.649104 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp" pod "2d591b68-3bbc-4432-9bf2-57a089da8872" (UID: "2d591b68-3bbc-4432-9bf2-57a089da8872"). InnerVolumeSpecName "kube-api-access-9fxtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:13.649842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.649823 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca" pod "2d591b68-3bbc-4432-9bf2-57a089da8872" (UID: "2d591b68-3bbc-4432-9bf2-57a089da8872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:13.748315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.748228 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxtp\" (UniqueName: \"kubernetes.io/projected/2d591b68-3bbc-4432-9bf2-57a089da8872-kube-api-access-9fxtp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:13.748315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.748268 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:13.748315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.748286 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d591b68-3bbc-4432-9bf2-57a089da8872-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:13.868516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.867114 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" event={"ID":"2d591b68-3bbc-4432-9bf2-57a089da8872","Type":"ContainerDied","Data":"fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0"} Nov 03 22:39:13.868516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.867199 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba17ad2aa85add934367b567b347196050fb71d72c8435a5b13fdb405a4abb0" Nov 03 22:39:13.868516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.867297 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk" Nov 03 22:39:13.891856 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:13.891458 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edd9358_77b3_431c_8f9a_58a32a85e585.slice/crio-e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447 WatchSource:0}: Error finding container e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447: Status 404 returned error can't find the container with id e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447 Nov 03 22:39:13.903773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:13.903389 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn"] Nov 03 22:39:14.493840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.493729 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:39:14.493840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.493787 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:39:14.498564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.498466 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:39:14.498564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.498515 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:39:14.501543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.501524 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:39:14.501685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.501668 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 22:39:14.890488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.890451 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" event={"ID":"3edd9358-77b3-431c-8f9a-58a32a85e585","Type":"ContainerStarted","Data":"8616d22878f20e966e5ea8b5d21e0841ded4508f003f36eb96a98d7f63ec691e"} Nov 03 22:39:14.891064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:14.891041 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" event={"ID":"3edd9358-77b3-431c-8f9a-58a32a85e585","Type":"ContainerStarted","Data":"e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447"} Nov 03 22:39:15.008357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.008299 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" podStartSLOduration=3.008271998 podStartE2EDuration="3.008271998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:15.005890326 +0000 UTC m=+1028.743234546" watchObservedRunningTime="2026-11-03 22:39:15.008271998 +0000 UTC m=+1028.745616224" Nov 03 22:39:15.795124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.792802 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw"] Nov 03 22:39:15.795124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.793778 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2d591b68-3bbc-4432-9bf2-57a089da8872" containerName="revoke-keys" Nov 03 22:39:15.795124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.793799 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d591b68-3bbc-4432-9bf2-57a089da8872" containerName="revoke-keys" Nov 03 22:39:15.795124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.793927 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d591b68-3bbc-4432-9bf2-57a089da8872" containerName="revoke-keys" Nov 03 22:39:15.816324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.816292 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:15.878013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.877132 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw"] Nov 03 22:39:15.970957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.970923 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:15.971635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.971608 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:15.971797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:15.971779 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzz9\" (UniqueName: \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.086105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.076046 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.086485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.086452 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.086634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.086619 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzz9\" (UniqueName: \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.096013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.087465 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.181360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.159950 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.192034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.182218 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzz9\" (UniqueName: \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\") pod \"maas-api-revoke-sub-e71852cd-k89tw\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.223442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.221740 5051 generic.go:356] "Generic (PLEG): container finished" podID="3edd9358-77b3-431c-8f9a-58a32a85e585" containerID="8616d22878f20e966e5ea8b5d21e0841ded4508f003f36eb96a98d7f63ec691e" exitCode=0 Nov 03 22:39:16.223442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.221957 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" event={"ID":"3edd9358-77b3-431c-8f9a-58a32a85e585","Type":"ContainerDied","Data":"8616d22878f20e966e5ea8b5d21e0841ded4508f003f36eb96a98d7f63ec691e"} Nov 03 22:39:16.347523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.347433 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:16.546518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.546394 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hn9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:39:16.546518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.546481 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:39:16.555194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.555142 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:39:16.555369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.555204 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:39:16.555566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.555540 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2jqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:39:16.555663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.555572 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:39:16.629221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.624847 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:16.629221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.626648 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:16.631537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.631513 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:39:16.644113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.643919 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:16.644693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.644434 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:16.647771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.647713 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:39:16.700762 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.700697 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:16.702651 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.701166 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:16.703258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.703213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" Nov 03 22:39:16.924507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:16.924477 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw"] Nov 03 22:39:16.925157 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:16.925127 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15eb386_f08d_4016_8966_b3611ed42380.slice/crio-90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9 WatchSource:0}: Error finding container 90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9: Status 404 returned error can't find the container with id 90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9 Nov 03 22:39:17.170001 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.169957 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:17.187650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.187581 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:17.235338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.235300 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" event={"ID":"a15eb386-f08d-4016-8966-b3611ed42380","Type":"ContainerStarted","Data":"90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9"} Nov 03 22:39:17.247075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.247048 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:39:17.390808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.390774 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:17.421794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.421755 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\") pod \"3edd9358-77b3-431c-8f9a-58a32a85e585\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " Nov 03 22:39:17.423297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.421825 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\") pod \"3edd9358-77b3-431c-8f9a-58a32a85e585\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " Nov 03 22:39:17.423297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.422073 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\" (UniqueName: \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\") pod \"3edd9358-77b3-431c-8f9a-58a32a85e585\" (UID: \"3edd9358-77b3-431c-8f9a-58a32a85e585\") " Nov 03 22:39:17.423297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.422402 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca" pod "3edd9358-77b3-431c-8f9a-58a32a85e585" (UID: "3edd9358-77b3-431c-8f9a-58a32a85e585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:17.423297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.422549 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:17.428408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.426406 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca" pod "3edd9358-77b3-431c-8f9a-58a32a85e585" (UID: "3edd9358-77b3-431c-8f9a-58a32a85e585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:17.428408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.427551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh" pod "3edd9358-77b3-431c-8f9a-58a32a85e585" (UID: "3edd9358-77b3-431c-8f9a-58a32a85e585"). InnerVolumeSpecName "kube-api-access-894rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:17.523492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.523458 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894rh\" (UniqueName: \"kubernetes.io/projected/3edd9358-77b3-431c-8f9a-58a32a85e585-kube-api-access-894rh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:17.523492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:17.523484 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edd9358-77b3-431c-8f9a-58a32a85e585-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:18.239396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.239357 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v"] Nov 03 22:39:18.240204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.240179 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3edd9358-77b3-431c-8f9a-58a32a85e585" containerName="revoke-keys" Nov 03 22:39:18.240204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.240203 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edd9358-77b3-431c-8f9a-58a32a85e585" containerName="revoke-keys" Nov 03 22:39:18.240412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.240370 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edd9358-77b3-431c-8f9a-58a32a85e585" containerName="revoke-keys" Nov 03 22:39:18.244983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.244949 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.250656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.250628 5051 generic.go:356] "Generic (PLEG): container finished" podID="a15eb386-f08d-4016-8966-b3611ed42380" containerID="8533a15597b6262553fc6bcbe3b76b9fc4080f76d7ec0961ac3658e580250d0e" exitCode=0 Nov 03 22:39:18.250772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.250676 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" event={"ID":"a15eb386-f08d-4016-8966-b3611ed42380","Type":"ContainerDied","Data":"8533a15597b6262553fc6bcbe3b76b9fc4080f76d7ec0961ac3658e580250d0e"} Nov 03 22:39:18.252536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.252503 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" Nov 03 22:39:18.252635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.252551 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn" event={"ID":"3edd9358-77b3-431c-8f9a-58a32a85e585","Type":"ContainerDied","Data":"e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447"} Nov 03 22:39:18.252690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.252630 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b85bbda5d08900d9facf40951dfedff415fa2ee85aab329791f0757f837447" Nov 03 22:39:18.315550 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.304230 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v"] Nov 03 22:39:18.340013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.332742 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.340013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.332791 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grhn\" (UniqueName: \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.340013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.333045 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.433924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.433892 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.434606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.434581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.434779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.434757 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grhn\" (UniqueName: \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.435460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.435437 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.437478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.437458 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.498430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.498396 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grhn\" (UniqueName: \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\") pod \"maas-api-revoke-sub-ae82e407-ztm2v\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:18.574063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:18.568203 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:19.066022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:19.065238 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v"] Nov 03 22:39:19.376248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:19.373548 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" event={"ID":"f0216f55-629c-4ecf-a004-92cb34f3d040","Type":"ContainerStarted","Data":"ef985d61dd8b5d76c2c04599d99ecf698061e3f5622ed41fa1ff5155ffbb1225"} Nov 03 22:39:20.174974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.174950 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:20.256228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.256191 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\" (UniqueName: \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\") pod \"a15eb386-f08d-4016-8966-b3611ed42380\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " Nov 03 22:39:20.256420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.256326 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\") pod \"a15eb386-f08d-4016-8966-b3611ed42380\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " Nov 03 22:39:20.256515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.256441 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\") pod \"a15eb386-f08d-4016-8966-b3611ed42380\" (UID: \"a15eb386-f08d-4016-8966-b3611ed42380\") " Nov 03 22:39:20.257310 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.257273 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca" pod "a15eb386-f08d-4016-8966-b3611ed42380" (UID: "a15eb386-f08d-4016-8966-b3611ed42380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:20.260956 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.260890 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9" pod "a15eb386-f08d-4016-8966-b3611ed42380" (UID: "a15eb386-f08d-4016-8966-b3611ed42380"). InnerVolumeSpecName "kube-api-access-9rzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:20.261186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.261160 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca" pod "a15eb386-f08d-4016-8966-b3611ed42380" (UID: "a15eb386-f08d-4016-8966-b3611ed42380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:20.358652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.358606 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzz9\" (UniqueName: \"kubernetes.io/projected/a15eb386-f08d-4016-8966-b3611ed42380-kube-api-access-9rzz9\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:20.358652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.358651 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:20.358833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.358669 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15eb386-f08d-4016-8966-b3611ed42380-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:20.384822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.384785 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" Nov 03 22:39:20.384822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.384807 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw" event={"ID":"a15eb386-f08d-4016-8966-b3611ed42380","Type":"ContainerDied","Data":"90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9"} Nov 03 22:39:20.385410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.384886 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d1a9f89ae9bdd243067e8eea034249692a8420f3049ee5919a261637c13fc9" Nov 03 22:39:20.386748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.386715 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" event={"ID":"f0216f55-629c-4ecf-a004-92cb34f3d040","Type":"ContainerStarted","Data":"9ee91d4116bc341dc90ff70f16fba4ee4ac2d09ae07401ae2a10bf1ecd42dd6d"} Nov 03 22:39:20.532116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:20.532017 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" podStartSLOduration=2.53198054 podStartE2EDuration="2.53198054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:20.529872939 +0000 UTC m=+1034.267217163" watchObservedRunningTime="2026-11-03 22:39:20.53198054 +0000 UTC m=+1034.269324764" Nov 03 22:39:21.398179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:21.398143 5051 generic.go:356] "Generic (PLEG): container finished" podID="f0216f55-629c-4ecf-a004-92cb34f3d040" containerID="9ee91d4116bc341dc90ff70f16fba4ee4ac2d09ae07401ae2a10bf1ecd42dd6d" exitCode=0 Nov 03 22:39:21.398587 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:21.398196 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" event={"ID":"f0216f55-629c-4ecf-a004-92cb34f3d040","Type":"ContainerDied","Data":"9ee91d4116bc341dc90ff70f16fba4ee4ac2d09ae07401ae2a10bf1ecd42dd6d"} Nov 03 22:39:21.624716 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:39:21.624677 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:39:22.580350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.580328 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:22.685534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.685459 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\" (UniqueName: \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\") pod \"f0216f55-629c-4ecf-a004-92cb34f3d040\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " Nov 03 22:39:22.685534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.685495 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\") pod \"f0216f55-629c-4ecf-a004-92cb34f3d040\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " Nov 03 22:39:22.685534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.685535 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\") pod \"f0216f55-629c-4ecf-a004-92cb34f3d040\" (UID: \"f0216f55-629c-4ecf-a004-92cb34f3d040\") " Nov 03 22:39:22.685927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.685895 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca" pod "f0216f55-629c-4ecf-a004-92cb34f3d040" (UID: "f0216f55-629c-4ecf-a004-92cb34f3d040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:22.686105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.686085 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:22.687877 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.687849 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca" pod "f0216f55-629c-4ecf-a004-92cb34f3d040" (UID: "f0216f55-629c-4ecf-a004-92cb34f3d040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:22.687964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.687916 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn" pod "f0216f55-629c-4ecf-a004-92cb34f3d040" (UID: "f0216f55-629c-4ecf-a004-92cb34f3d040"). InnerVolumeSpecName "kube-api-access-4grhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:22.786829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.786795 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grhn\" (UniqueName: \"kubernetes.io/projected/f0216f55-629c-4ecf-a004-92cb34f3d040-kube-api-access-4grhn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:22.786829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:22.786824 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0216f55-629c-4ecf-a004-92cb34f3d040-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:23.420764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:23.420726 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" event={"ID":"f0216f55-629c-4ecf-a004-92cb34f3d040","Type":"ContainerDied","Data":"ef985d61dd8b5d76c2c04599d99ecf698061e3f5622ed41fa1ff5155ffbb1225"} Nov 03 22:39:23.420764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:23.420749 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v" Nov 03 22:39:23.420948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:23.420780 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef985d61dd8b5d76c2c04599d99ecf698061e3f5622ed41fa1ff5155ffbb1225" Nov 03 22:39:26.548644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.548615 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9"] Nov 03 22:39:26.549184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549171 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f0216f55-629c-4ecf-a004-92cb34f3d040" containerName="revoke-keys" Nov 03 22:39:26.549225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549186 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0216f55-629c-4ecf-a004-92cb34f3d040" containerName="revoke-keys" Nov 03 22:39:26.549256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549225 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a15eb386-f08d-4016-8966-b3611ed42380" containerName="revoke-keys" Nov 03 22:39:26.549256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549231 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15eb386-f08d-4016-8966-b3611ed42380" containerName="revoke-keys" Nov 03 22:39:26.549328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549317 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0216f55-629c-4ecf-a004-92cb34f3d040" containerName="revoke-keys" Nov 03 22:39:26.549359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.549328 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15eb386-f08d-4016-8966-b3611ed42380" containerName="revoke-keys" Nov 03 22:39:26.553098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.553084 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.610658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.610626 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9"] Nov 03 22:39:26.636300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.636266 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.636503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.636328 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.636581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.636562 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhsn\" (UniqueName: \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.654520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.654489 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:39:26.654675 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.654572 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:39:26.737475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.737432 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.737664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.737573 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhsn\" (UniqueName: \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.737664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.737634 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.738274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.738255 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.740031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.740009 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.798124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.798096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhsn\" (UniqueName: \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\") pod \"maas-api-revoke-sub-fd1a0bb9-6wtc9\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:26.866827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:26.866803 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:27.247259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:27.245926 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9"] Nov 03 22:39:27.247456 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:27.247428 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8287ed12_7127_4433_bf00_96e28dac4ee5.slice/crio-93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723 WatchSource:0}: Error finding container 93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723: Status 404 returned error can't find the container with id 93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723 Nov 03 22:39:27.451568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:27.451531 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" event={"ID":"8287ed12-7127-4433-bf00-96e28dac4ee5","Type":"ContainerStarted","Data":"93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723"} Nov 03 22:39:28.462871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:28.462776 5051 generic.go:356] "Generic (PLEG): container finished" podID="8287ed12-7127-4433-bf00-96e28dac4ee5" containerID="cf33a7cfd0838e5150ac6cac51506213816930aab5dc2bfa641a9f4963987a1d" exitCode=0 Nov 03 22:39:28.462871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:28.462821 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" event={"ID":"8287ed12-7127-4433-bf00-96e28dac4ee5","Type":"ContainerDied","Data":"cf33a7cfd0838e5150ac6cac51506213816930aab5dc2bfa641a9f4963987a1d"} Nov 03 22:39:29.646295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.646270 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:29.775807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.775718 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\" (UniqueName: \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\") pod \"8287ed12-7127-4433-bf00-96e28dac4ee5\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " Nov 03 22:39:29.775979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.775906 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\") pod \"8287ed12-7127-4433-bf00-96e28dac4ee5\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " Nov 03 22:39:29.775979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.775955 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\") pod \"8287ed12-7127-4433-bf00-96e28dac4ee5\" (UID: \"8287ed12-7127-4433-bf00-96e28dac4ee5\") " Nov 03 22:39:29.776533 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.776499 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca" pod "8287ed12-7127-4433-bf00-96e28dac4ee5" (UID: "8287ed12-7127-4433-bf00-96e28dac4ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:29.784030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.778774 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca" pod "8287ed12-7127-4433-bf00-96e28dac4ee5" (UID: "8287ed12-7127-4433-bf00-96e28dac4ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:29.784030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.780144 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn" pod "8287ed12-7127-4433-bf00-96e28dac4ee5" (UID: "8287ed12-7127-4433-bf00-96e28dac4ee5"). InnerVolumeSpecName "kube-api-access-xzhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:29.877591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.877559 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhsn\" (UniqueName: \"kubernetes.io/projected/8287ed12-7127-4433-bf00-96e28dac4ee5-kube-api-access-xzhsn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:29.877591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.877589 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:29.877591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:29.877600 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8287ed12-7127-4433-bf00-96e28dac4ee5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:30.480603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:30.480572 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" Nov 03 22:39:30.480603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:30.480585 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9" event={"ID":"8287ed12-7127-4433-bf00-96e28dac4ee5","Type":"ContainerDied","Data":"93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723"} Nov 03 22:39:30.480797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:30.480631 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93823d5799db62ff6593fbae480d736e6f4f7c2f843c9d477629eda6a6ffb723" Nov 03 22:39:42.375051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.375015 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm"] Nov 03 22:39:42.375599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.375586 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8287ed12-7127-4433-bf00-96e28dac4ee5" containerName="revoke-keys" Nov 03 22:39:42.375646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.375600 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8287ed12-7127-4433-bf00-96e28dac4ee5" containerName="revoke-keys" Nov 03 22:39:42.375716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.375707 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8287ed12-7127-4433-bf00-96e28dac4ee5" containerName="revoke-keys" Nov 03 22:39:42.379306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.379290 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.436374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.436343 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm"] Nov 03 22:39:42.480583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.480550 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:39:42.480751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.480666 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:39:42.554451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.554421 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.554635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.554456 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.554635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.554492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvz9x\" (UniqueName: \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.656344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.656257 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.656344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.656297 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.656344 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.656331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvz9x\" (UniqueName: \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.657101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.657076 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.659024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.658978 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.717294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.717265 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvz9x\" (UniqueName: \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\") pod \"maas-api-revoke-sub-338b4283-pqggm\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:42.993537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:42.993457 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:43.376982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:43.376953 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm"] Nov 03 22:39:43.377890 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:43.377862 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6b8c88_1459_4e77_8ca2_3ed3eb131507.slice/crio-ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac WatchSource:0}: Error finding container ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac: Status 404 returned error can't find the container with id ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac Nov 03 22:39:43.577202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:43.577168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" event={"ID":"2f6b8c88-1459-4e77-8ca2-3ed3eb131507","Type":"ContainerStarted","Data":"ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac"} Nov 03 22:39:44.585657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:44.585558 5051 generic.go:356] "Generic (PLEG): container finished" podID="2f6b8c88-1459-4e77-8ca2-3ed3eb131507" containerID="7d12796f8af107b94b539dc33615274783e043f6928d33321f1f9cd2d760fcc3" exitCode=0 Nov 03 22:39:44.585657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:44.585596 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" event={"ID":"2f6b8c88-1459-4e77-8ca2-3ed3eb131507","Type":"ContainerDied","Data":"7d12796f8af107b94b539dc33615274783e043f6928d33321f1f9cd2d760fcc3"} Nov 03 22:39:45.761612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.761588 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:45.797012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.796969 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\") pod \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " Nov 03 22:39:45.797189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.797042 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\" (UniqueName: \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\") pod \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " Nov 03 22:39:45.797332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.797306 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca" pod "2f6b8c88-1459-4e77-8ca2-3ed3eb131507" (UID: "2f6b8c88-1459-4e77-8ca2-3ed3eb131507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:45.797397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.797333 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\") pod \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\" (UID: \"2f6b8c88-1459-4e77-8ca2-3ed3eb131507\") " Nov 03 22:39:45.797818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.797794 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:45.799328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.799303 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x" pod "2f6b8c88-1459-4e77-8ca2-3ed3eb131507" (UID: "2f6b8c88-1459-4e77-8ca2-3ed3eb131507"). InnerVolumeSpecName "kube-api-access-dvz9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:45.799399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.799386 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca" pod "2f6b8c88-1459-4e77-8ca2-3ed3eb131507" (UID: "2f6b8c88-1459-4e77-8ca2-3ed3eb131507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:45.899452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.899419 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:45.899452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:45.899453 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvz9x\" (UniqueName: \"kubernetes.io/projected/2f6b8c88-1459-4e77-8ca2-3ed3eb131507-kube-api-access-dvz9x\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:39:46.604156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:46.604124 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" Nov 03 22:39:46.604337 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:46.604160 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm" event={"ID":"2f6b8c88-1459-4e77-8ca2-3ed3eb131507","Type":"ContainerDied","Data":"ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac"} Nov 03 22:39:46.604337 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:46.604206 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac57f7ea2463c046b2375b9a10fc435945d159754bd9cb799488487d47b1e1ac" Nov 03 22:39:51.623054 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:39:51.623021 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:39:57.695514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.695478 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 22:39:57.698048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.696234 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2f6b8c88-1459-4e77-8ca2-3ed3eb131507" containerName="revoke-keys" Nov 03 22:39:57.698048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.696249 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6b8c88-1459-4e77-8ca2-3ed3eb131507" containerName="revoke-keys" Nov 03 22:39:57.698048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.696374 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6b8c88-1459-4e77-8ca2-3ed3eb131507" containerName="revoke-keys" Nov 03 22:39:57.699718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.699701 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.757265 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.757233 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 22:39:57.801743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.801715 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"distinct-w2-164c36-kserve-self-signed-certs\"" Nov 03 22:39:57.801936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.801805 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"default-dockercfg-rb9gv\"" Nov 03 22:39:57.801936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.801823 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"openshift-service-ca.crt\"" Nov 03 22:39:57.801936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.801876 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"kube-root-ca.crt\"" Nov 03 22:39:57.851150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851024 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.851150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851090 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.851455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851366 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvrwn\" (UniqueName: \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.851455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851440 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.851574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851501 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.851574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.851531 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvrwn\" (UniqueName: \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952646 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952699 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952722 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952824 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.952917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.952856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.953160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.953139 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.953201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.953182 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.953235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.953199 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.955419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.955398 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:57.955634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:57.955619 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:58.013558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.013515 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvrwn\" (UniqueName: \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\") pod \"distinct-w2-164c36-kserve-74ccd44b76-vk55q\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:58.013745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.013726 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:39:58.413236 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.413213 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 22:39:58.414299 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:39:58.414278 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1027699_4263_49e9_b2db_c009f17b57cd.slice/crio-0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07 WatchSource:0}: Error finding container 0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07: Status 404 returned error can't find the container with id 0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07 Nov 03 22:39:58.698086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.697959 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" event={"ID":"b1027699-4263-49e9-b2db-c009f17b57cd","Type":"ContainerStarted","Data":"cd2d8c02f2f3e92eeb20e811cdc592e0588ef187a72026b843e12f721d0d5ab0"} Nov 03 22:39:58.698086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.698026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" event={"ID":"b1027699-4263-49e9-b2db-c009f17b57cd","Type":"ContainerStarted","Data":"0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07"} Nov 03 22:39:58.814643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:58.814590 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podStartSLOduration=1.8145751159999999 podStartE2EDuration="1.814575116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:58.813850754 +0000 UTC m=+1072.551194978" watchObservedRunningTime="2026-11-03 22:39:58.814575116 +0000 UTC m=+1072.551919340" Nov 03 22:39:59.014305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:39:59.014226 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:40:08.014957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:08.014924 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:40:08.027469 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:08.027446 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:40:08.799805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:08.799773 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:40:16.206049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.206015 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 22:40:16.211437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.211421 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.292802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.292763 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.293031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.292885 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.293031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.292944 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.293031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.292983 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6dw\" (UniqueName: \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.293224 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.293064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.293224 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.293151 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.304692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.304661 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 22:40:16.317764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.317728 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-32449d\"/\"distinct-w4-32449d-kserve-self-signed-certs\"" Nov 03 22:40:16.394345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394306 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.394576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394389 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.394576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394433 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6dw\" (UniqueName: \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.394576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394494 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.394722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394582 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.394722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.394690 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.395249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.395220 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.396312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.396287 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.396450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.396308 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.404945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.404518 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.405645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.405537 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.456455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.456389 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6dw\" (UniqueName: \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\") pod \"distinct-w4-32449d-kserve-6997c4579c-8wqrv\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.523392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.523364 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:16.947209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:16.947178 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 22:40:16.947622 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:40:16.947596 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46dfef3f_5d82_4eed_b46e_4c7c0c5a40c2.slice/crio-1ee6fe79280e1ff0387f2a23aee5ef9d5d95438cf1e0700923ed49afb3e5c8cf WatchSource:0}: Error finding container 1ee6fe79280e1ff0387f2a23aee5ef9d5d95438cf1e0700923ed49afb3e5c8cf: Status 404 returned error can't find the container with id 1ee6fe79280e1ff0387f2a23aee5ef9d5d95438cf1e0700923ed49afb3e5c8cf Nov 03 22:40:17.857780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:17.857739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" event={"ID":"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2","Type":"ContainerStarted","Data":"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3"} Nov 03 22:40:17.858321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:17.857789 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" event={"ID":"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2","Type":"ContainerStarted","Data":"1ee6fe79280e1ff0387f2a23aee5ef9d5d95438cf1e0700923ed49afb3e5c8cf"} Nov 03 22:40:17.977277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:17.977223 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podStartSLOduration=1.9772091 podStartE2EDuration="1.9772091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:17.97558787 +0000 UTC m=+1091.712932098" watchObservedRunningTime="2026-11-03 22:40:17.9772091 +0000 UTC m=+1091.714553324" Nov 03 22:40:18.523979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:18.523948 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:20.265482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.265441 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 22:40:20.269668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.269647 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.345654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.345623 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.345839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.345759 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.345910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.345853 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.345910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.345887 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.346129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.346047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.346129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.346091 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5qt\" (UniqueName: \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.371213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.371189 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7540da\"/\"openshift-service-ca.crt\"" Nov 03 22:40:20.371213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.371197 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7540da\"/\"kube-root-ca.crt\"" Nov 03 22:40:20.371411 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.371392 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7540da\"/\"default-dockercfg-7hxtx\"" Nov 03 22:40:20.371469 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.371418 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7540da\"/\"distinct-w6-7540da-kserve-self-signed-certs\"" Nov 03 22:40:20.375588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.375552 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.447705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.447751 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5qt\" (UniqueName: \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.447911 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.448436 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.448564 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.448583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.448719 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.448768 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.449261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.449163 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.452097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.452053 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.452886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.452862 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.514218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.514188 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5qt\" (UniqueName: \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\") pod \"distinct-w6-7540da-kserve-84d988f488-9h9pt\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.583229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.583152 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:20.988714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:20.988685 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 22:40:20.989826 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:40:20.989796 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0103607_e3d9_4027_8398_657c568682ba.slice/crio-99586abf97fc872a0b6068ba5cc439f86a5f19f8386b8e479d140d5447cfd73d WatchSource:0}: Error finding container 99586abf97fc872a0b6068ba5cc439f86a5f19f8386b8e479d140d5447cfd73d: Status 404 returned error can't find the container with id 99586abf97fc872a0b6068ba5cc439f86a5f19f8386b8e479d140d5447cfd73d Nov 03 22:40:21.624693 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:40:21.624652 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:40:21.887960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:21.887872 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" event={"ID":"d0103607-e3d9-4027-8398-657c568682ba","Type":"ContainerStarted","Data":"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb"} Nov 03 22:40:21.887960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:21.887919 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" event={"ID":"d0103607-e3d9-4027-8398-657c568682ba","Type":"ContainerStarted","Data":"99586abf97fc872a0b6068ba5cc439f86a5f19f8386b8e479d140d5447cfd73d"} Nov 03 22:40:22.007387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:22.007343 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podStartSLOduration=2.007329884 podStartE2EDuration="2.007329884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:22.003207913 +0000 UTC m=+1095.740552329" watchObservedRunningTime="2026-11-03 22:40:22.007329884 +0000 UTC m=+1095.744674109" Nov 03 22:40:22.583377 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:22.583341 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:26.524528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:26.524497 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:26.537186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:26.537159 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:26.937511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:26.937478 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:40:27.108843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.108812 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-98mdw-kfggc"] Nov 03 22:40:27.113088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.113063 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.165949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.165889 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cbh97\"" Nov 03 22:40:27.173935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.173904 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-98mdw-kfggc"] Nov 03 22:40:27.222971 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.222878 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.222971 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.222925 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.223219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.223063 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.223273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.223257 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvrb\" (UniqueName: \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324009 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324060 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324098 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324240 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvrb\" (UniqueName: \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324466 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.324797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.324777 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.325109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.325091 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.386240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.386203 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvrb\" (UniqueName: \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\") pod \"periodic-gathering-98mdw-kfggc\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.429645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.429604 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:40:27.868639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.868614 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-98mdw-kfggc"] Nov 03 22:40:27.869187 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:40:27.869159 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f40dc2c_19c3_4e08_95c4_edcf98c10a01.slice/crio-5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772 WatchSource:0}: Error finding container 5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772: Status 404 returned error can't find the container with id 5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772 Nov 03 22:40:27.935519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:27.935486 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-98mdw-kfggc" event={"ID":"4f40dc2c-19c3-4e08-95c4-edcf98c10a01","Type":"ContainerStarted","Data":"5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772"} Nov 03 22:40:28.955120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:28.955080 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-98mdw-kfggc" event={"ID":"4f40dc2c-19c3-4e08-95c4-edcf98c10a01","Type":"ContainerStarted","Data":"dbec321f583c5aa2d2d1149776f9762aa30c463462bbee1aa2b2f27e67666506"} Nov 03 22:40:29.071312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:29.071257 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-98mdw-kfggc" podStartSLOduration=2.071242775 podStartE2EDuration="2.071242775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:29.069210351 +0000 UTC m=+1102.806554576" watchObservedRunningTime="2026-11-03 22:40:29.071242775 +0000 UTC m=+1102.808587000" Nov 03 22:40:30.583414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:30.583378 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:30.597717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:30.597687 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:30.999175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:30.999141 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:40:33.242717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.242681 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:40:33.247252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.247235 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.348688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.348657 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:40:33.413245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413209 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413285 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413397 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413474 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413523 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglc9\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413544 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.413646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.413568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.514901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.514810 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglc9\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.514901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.514859 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515146 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.514932 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515184 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515252 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515327 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515364 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.515756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515724 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.516010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.515965 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.516286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.516171 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.516286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.516245 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.516625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.516603 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.518520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.518483 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.519355 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.519333 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.575707 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.575672 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.575959 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.575936 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglc9\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:33.861081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:33.861040 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:34.304343 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:40:34.304303 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff874b2_62ba_4c52_9183_843cfd37a3d0.slice/crio-e73f0b5eb0a2a490b7f2af2b4689d68a56d9f0beb185f44da1415b789b6dff56 WatchSource:0}: Error finding container e73f0b5eb0a2a490b7f2af2b4689d68a56d9f0beb185f44da1415b789b6dff56: Status 404 returned error can't find the container with id e73f0b5eb0a2a490b7f2af2b4689d68a56d9f0beb185f44da1415b789b6dff56 Nov 03 22:40:34.305779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:34.305752 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:40:34.307674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:34.307638 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:34.307792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:34.307717 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:34.307859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:34.307788 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:35.023287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.023251 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" event={"ID":"9ff874b2-62ba-4c52-9183-843cfd37a3d0","Type":"ContainerStarted","Data":"1779609fc8157c3d31cd0f2208fcad51de087678ced13de42254c5db3a7a3e46"} Nov 03 22:40:35.023461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.023317 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" event={"ID":"9ff874b2-62ba-4c52-9183-843cfd37a3d0","Type":"ContainerStarted","Data":"e73f0b5eb0a2a490b7f2af2b4689d68a56d9f0beb185f44da1415b789b6dff56"} Nov 03 22:40:35.250124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.250076 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podStartSLOduration=2.2500622789999998 podStartE2EDuration="2.250062279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:35.195799154 +0000 UTC m=+1108.933143380" watchObservedRunningTime="2026-11-03 22:40:35.250062279 +0000 UTC m=+1108.987406500" Nov 03 22:40:35.864645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.864611 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:35.865278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.865256 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:35.865429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:35.865409 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:36.880016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:36.878775 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:36.880016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:36.878842 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:37.864015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:37.863373 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:37.864015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:37.863437 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:38.875016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:38.865314 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:38.875016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:38.865373 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:39.867329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:39.867293 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:39.867582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:39.867552 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:40.861955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:40.861912 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:40:40.862398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:40.862023 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:40:41.867150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:41.867123 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:41.867640 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:41.867414 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:41.868307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:41.868286 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:40:42.445495 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.445464 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 22:40:42.450082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.450063 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.503157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.503129 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"distinct-w1-7afb1c-kserve-self-signed-certs\"" Nov 03 22:40:42.512335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.512302 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 22:40:42.524407 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524366 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vww7p\" (UniqueName: \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.524538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524425 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.524588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524560 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.524635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524590 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.524635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524614 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.524751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.524736 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626388 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vww7p\" (UniqueName: \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626512 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626535 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626751 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.626897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626872 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.627011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626964 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.627098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.626976 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.627359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.627333 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.630166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.630139 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.631964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.631937 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.687171 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.687141 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vww7p\" (UniqueName: \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\") pod \"distinct-w1-7afb1c-kserve-56cc87768-lq48w\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:42.764039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:42.763938 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:43.172961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:43.172928 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 22:40:43.173427 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:40:43.173205 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5ab1960_193f_449d_b053_3e559a376825.slice/crio-ae1684829915334337f364bf602ac4af0cf791a7ea99bc0b429b8a43c63f5ec6 WatchSource:0}: Error finding container ae1684829915334337f364bf602ac4af0cf791a7ea99bc0b429b8a43c63f5ec6: Status 404 returned error can't find the container with id ae1684829915334337f364bf602ac4af0cf791a7ea99bc0b429b8a43c63f5ec6 Nov 03 22:40:43.217070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:43.217039 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" event={"ID":"c5ab1960-193f-449d-b053-3e559a376825","Type":"ContainerStarted","Data":"ae1684829915334337f364bf602ac4af0cf791a7ea99bc0b429b8a43c63f5ec6"} Nov 03 22:40:44.225951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:44.225914 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" event={"ID":"c5ab1960-193f-449d-b053-3e559a376825","Type":"ContainerStarted","Data":"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382"} Nov 03 22:40:44.359611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:44.359554 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podStartSLOduration=2.359541591 podStartE2EDuration="2.359541591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:44.357406852 +0000 UTC m=+1118.094751100" watchObservedRunningTime="2026-11-03 22:40:44.359541591 +0000 UTC m=+1118.096885808" Nov 03 22:40:44.764378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:44.764345 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:51.624183 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:40:51.624102 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:40:52.764889 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:52.764855 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:52.777504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:52.777479 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:40:53.306629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:40:53.306600 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:41:02.114383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.114347 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:41:02.120685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.120659 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqsm\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211458 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211504 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211567 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211625 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211658 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.211808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.211702 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313433 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqsm\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313517 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313546 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313585 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313635 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313703 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313763 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313802 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.313848 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.314407 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.315245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.314969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.316043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.315944 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.316257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.316231 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.316850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.316812 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.318158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.318135 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.319086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.319055 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.329371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.329346 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:41:02.376027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.375920 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqsm\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.376235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.376211 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.436304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.436275 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:02.875802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.875768 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:41:02.876954 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:41:02.876919 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eff83a7_5ef9_4f6c_9949_2202d18946e3.slice/crio-d3b65f70c68fba2d7e312551a6f8d6da0bf2d78fc342ed555cca3049f57015b1 WatchSource:0}: Error finding container d3b65f70c68fba2d7e312551a6f8d6da0bf2d78fc342ed555cca3049f57015b1: Status 404 returned error can't find the container with id d3b65f70c68fba2d7e312551a6f8d6da0bf2d78fc342ed555cca3049f57015b1 Nov 03 22:41:02.879740 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.879702 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:02.879841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.879784 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:02.879841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:02.879829 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:03.167681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.167599 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:41:03.173272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.173250 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.237974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.237935 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.238306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.238285 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.238485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.238468 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.238641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.238622 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.238768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.238754 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.238898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.238882 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.239036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.239020 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.239185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.239169 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.239351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.239337 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz4d\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.257803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.257771 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.341761 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.341857 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.341930 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342004 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342045 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342090 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342131 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342179 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342264 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz4d\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.342853 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.343102 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.343887 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.344141 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.345122 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.353915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.346408 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.369827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.369598 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.397910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.396961 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" event={"ID":"5eff83a7-5ef9-4f6c-9949-2202d18946e3","Type":"ContainerStarted","Data":"f3e6cf77a9dbfec6f2254068e7f212136ecc1bf4155363fa0a8da607f3bb9345"} Nov 03 22:41:03.397910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.397064 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" event={"ID":"5eff83a7-5ef9-4f6c-9949-2202d18946e3","Type":"ContainerStarted","Data":"d3b65f70c68fba2d7e312551a6f8d6da0bf2d78fc342ed555cca3049f57015b1"} Nov 03 22:41:03.405620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.403158 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz4d\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.406201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.406176 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.438088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.437477 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:03.489915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.489887 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:03.525586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:03.525521 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podStartSLOduration=1.525500477 podStartE2EDuration="1.525500477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:03.525295712 +0000 UTC m=+1137.262639938" watchObservedRunningTime="2026-11-03 22:41:03.525500477 +0000 UTC m=+1137.262844705" Nov 03 22:41:04.048596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.048564 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:41:04.092657 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:41:04.092604 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9601351_e31d_4fd0_9361_4859651458e8.slice/crio-08e104aee45e44b6f4d2e87715130521d16a44020258bf36afc22797c452cc20 WatchSource:0}: Error finding container 08e104aee45e44b6f4d2e87715130521d16a44020258bf36afc22797c452cc20: Status 404 returned error can't find the container with id 08e104aee45e44b6f4d2e87715130521d16a44020258bf36afc22797c452cc20 Nov 03 22:41:04.105564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.105503 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:04.105700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.105592 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:04.105700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.105633 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:04.420057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.420023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" event={"ID":"c9601351-e31d-4fd0-9361-4859651458e8","Type":"ContainerStarted","Data":"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d"} Nov 03 22:41:04.420559 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.420085 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" event={"ID":"c9601351-e31d-4fd0-9361-4859651458e8","Type":"ContainerStarted","Data":"08e104aee45e44b6f4d2e87715130521d16a44020258bf36afc22797c452cc20"} Nov 03 22:41:04.439092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.439063 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:04.439258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.439138 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:04.490201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.490167 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:04.552518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:04.552452 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podStartSLOduration=1.552433416 podStartE2EDuration="1.552433416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:04.550116703 +0000 UTC m=+1138.287460929" watchObservedRunningTime="2026-11-03 22:41:04.552433416 +0000 UTC m=+1138.289777643" Nov 03 22:41:05.441623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:05.441585 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:05.442276 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:05.442248 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:05.502322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:05.502288 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:05.502644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:05.502615 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:06.437266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:06.437239 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:06.437500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:06.437481 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:06.505786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:06.505744 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:06.509391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:06.505808 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:07.437297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:07.437262 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:07.437490 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:07.437315 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:07.494014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:07.490596 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:07.494014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:07.490682 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:08.436918 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:08.436884 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:08.437427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:08.437402 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:08.503930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:08.503894 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:08.504236 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:08.504206 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:09.436923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:09.436892 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:41:09.437349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:09.436951 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 22:41:09.490925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:09.490886 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:09.491119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:09.490955 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:10.441403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:10.441371 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:10.441802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:10.441783 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:10.442440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:10.442419 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:41:10.491190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:10.491161 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:41:10.491367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:10.491221 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 22:41:11.497599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:11.497560 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:11.498100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:11.497958 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:11.498707 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:11.498687 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:41:21.624985 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:41:21.624950 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:41:51.623960 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:41:51.623921 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:41:59.406175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.406141 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n"] Nov 03 22:41:59.410518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.410494 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.463021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.462924 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:59.468475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.468449 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n"] Nov 03 22:41:59.511708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.511679 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:41:59.511708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.511680 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.511945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.511928 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpskk\" (UniqueName: \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.512042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.512020 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.613438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.613405 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpskk\" (UniqueName: \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.613620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.613451 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.613620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.613515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.614113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.614093 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.615969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.615939 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.674324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.674256 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpskk\" (UniqueName: \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\") pod \"maas-api-revoke-sub-07e0f07d-rct6n\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:41:59.726034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:41:59.725980 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:42:00.115011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:00.114972 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n"] Nov 03 22:42:00.116028 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:42:00.115967 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a709e5_55e0_445e_a974_c368be486346.slice/crio-6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe WatchSource:0}: Error finding container 6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe: Status 404 returned error can't find the container with id 6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe Nov 03 22:42:00.955364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:00.955321 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" event={"ID":"70a709e5-55e0-445e-a974-c368be486346","Type":"ContainerStarted","Data":"4cd44b8efa944d73a361f4be087e732a619300d5e0e87cc70fd48359390c41c6"} Nov 03 22:42:00.956081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:00.955409 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" event={"ID":"70a709e5-55e0-445e-a974-c368be486346","Type":"ContainerStarted","Data":"6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe"} Nov 03 22:42:01.075184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:01.075134 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" podStartSLOduration=2.075119233 podStartE2EDuration="2.075119233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:01.069799622 +0000 UTC m=+1194.807143848" watchObservedRunningTime="2026-11-03 22:42:01.075119233 +0000 UTC m=+1194.812463460" Nov 03 22:42:01.965046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:01.964977 5051 generic.go:356] "Generic (PLEG): container finished" podID="70a709e5-55e0-445e-a974-c368be486346" containerID="4cd44b8efa944d73a361f4be087e732a619300d5e0e87cc70fd48359390c41c6" exitCode=0 Nov 03 22:42:01.965046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:01.965027 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" event={"ID":"70a709e5-55e0-445e-a974-c368be486346","Type":"ContainerDied","Data":"4cd44b8efa944d73a361f4be087e732a619300d5e0e87cc70fd48359390c41c6"} Nov 03 22:42:02.062145 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:02.062112 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w"] Nov 03 22:42:02.071143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:02.071115 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42117eca-mq98w"] Nov 03 22:42:02.412502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:02.412470 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b465d988-f46c-4d9e-83a8-a1e5f1da4eec" path="/var/lib/kubelet/pods/b465d988-f46c-4d9e-83a8-a1e5f1da4eec/volumes" Nov 03 22:42:03.554453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.554424 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:42:03.638768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.638728 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\" (UniqueName: \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\") pod \"70a709e5-55e0-445e-a974-c368be486346\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " Nov 03 22:42:03.638960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.638810 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\") pod \"70a709e5-55e0-445e-a974-c368be486346\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " Nov 03 22:42:03.638960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.638888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\") pod \"70a709e5-55e0-445e-a974-c368be486346\" (UID: \"70a709e5-55e0-445e-a974-c368be486346\") " Nov 03 22:42:03.639711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.639672 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca" pod "70a709e5-55e0-445e-a974-c368be486346" (UID: "70a709e5-55e0-445e-a974-c368be486346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:03.640739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.640719 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a709e5-55e0-445e-a974-c368be486346-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:03.642322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.642276 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk" pod "70a709e5-55e0-445e-a974-c368be486346" (UID: "70a709e5-55e0-445e-a974-c368be486346"). InnerVolumeSpecName "kube-api-access-jpskk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:03.642443 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.642360 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca" pod "70a709e5-55e0-445e-a974-c368be486346" (UID: "70a709e5-55e0-445e-a974-c368be486346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:03.742149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.742072 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpskk\" (UniqueName: \"kubernetes.io/projected/70a709e5-55e0-445e-a974-c368be486346-kube-api-access-jpskk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:03.742149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:03.742107 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a709e5-55e0-445e-a974-c368be486346-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:04.016334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:04.016303 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" Nov 03 22:42:04.016466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:04.016341 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n" event={"ID":"70a709e5-55e0-445e-a974-c368be486346","Type":"ContainerDied","Data":"6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe"} Nov 03 22:42:04.016466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:04.016398 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5496f8dddca07bd467bce5cadaa2620448f64069f5c1ec7b92cb37b7a15afe" Nov 03 22:42:04.018222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:04.018192 5051 generic.go:356] "Generic (PLEG): container finished" podID="4f40dc2c-19c3-4e08-95c4-edcf98c10a01" containerID="dbec321f583c5aa2d2d1149776f9762aa30c463462bbee1aa2b2f27e67666506" exitCode=0 Nov 03 22:42:04.018353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:04.018267 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-98mdw-kfggc" event={"ID":"4f40dc2c-19c3-4e08-95c4-edcf98c10a01","Type":"ContainerDied","Data":"dbec321f583c5aa2d2d1149776f9762aa30c463462bbee1aa2b2f27e67666506"} Nov 03 22:42:05.196666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.196639 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:42:05.259368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259337 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\") pod \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " Nov 03 22:42:05.259522 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259432 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\") pod \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " Nov 03 22:42:05.259608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259532 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\" (UniqueName: \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\") pod \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " Nov 03 22:42:05.259608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259581 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\") pod \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\" (UID: \"4f40dc2c-19c3-4e08-95c4-edcf98c10a01\") " Nov 03 22:42:05.259913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259793 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle" pod "4f40dc2c-19c3-4e08-95c4-edcf98c10a01" (UID: "4f40dc2c-19c3-4e08-95c4-edcf98c10a01"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:05.259913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259901 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle" pod "4f40dc2c-19c3-4e08-95c4-edcf98c10a01" (UID: "4f40dc2c-19c3-4e08-95c4-edcf98c10a01"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:05.260052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.259971 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path" pod "4f40dc2c-19c3-4e08-95c4-edcf98c10a01" (UID: "4f40dc2c-19c3-4e08-95c4-edcf98c10a01"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:05.260157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.260138 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-service-ca-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:05.260230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.260161 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-trusted-ca-bundle\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:05.260230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.260171 5051 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-archives-path\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:05.261872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.261849 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb" pod "4f40dc2c-19c3-4e08-95c4-edcf98c10a01" (UID: "4f40dc2c-19c3-4e08-95c4-edcf98c10a01"). InnerVolumeSpecName "kube-api-access-rdvrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:05.361434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:05.361395 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvrb\" (UniqueName: \"kubernetes.io/projected/4f40dc2c-19c3-4e08-95c4-edcf98c10a01-kube-api-access-rdvrb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:06.034026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:06.034001 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-98mdw-kfggc" Nov 03 22:42:06.034214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:06.034027 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-98mdw-kfggc" event={"ID":"4f40dc2c-19c3-4e08-95c4-edcf98c10a01","Type":"ContainerDied","Data":"5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772"} Nov 03 22:42:06.034214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:06.034076 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5796f5570bef0337b3b0dafd765e999e776da9e6220394f2351420040b5e0772" Nov 03 22:42:06.644902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:06.644874 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:42:06.645401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:06.645379 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:42:21.624515 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:42:21.624435 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:42:24.119401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.119370 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x"] Nov 03 22:42:24.120033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120018 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4f40dc2c-19c3-4e08-95c4-edcf98c10a01" containerName="insights-gathering" Nov 03 22:42:24.120083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120033 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f40dc2c-19c3-4e08-95c4-edcf98c10a01" containerName="insights-gathering" Nov 03 22:42:24.120083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120044 5051 cpu_manager.go:414] "containerMap: removing container" podUID="70a709e5-55e0-445e-a974-c368be486346" containerName="revoke-keys" Nov 03 22:42:24.120083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120049 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a709e5-55e0-445e-a974-c368be486346" containerName="revoke-keys" Nov 03 22:42:24.120178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120160 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f40dc2c-19c3-4e08-95c4-edcf98c10a01" containerName="insights-gathering" Nov 03 22:42:24.120178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.120172 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a709e5-55e0-445e-a974-c368be486346" containerName="revoke-keys" Nov 03 22:42:24.123728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.123710 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.182119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.182073 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x"] Nov 03 22:42:24.224893 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.224866 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:24.225056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.224893 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:42:24.305023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.304953 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.305190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.305156 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbbm\" (UniqueName: \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.305280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.305261 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.406538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.406455 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbbm\" (UniqueName: \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.406708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.406552 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.406708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.406639 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.407248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.407229 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.409291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.409270 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.467007 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.466969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbbm\" (UniqueName: \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\") pod \"maas-api-revoke-sub-45a9ec0a-ffj6x\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:24.737696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:24.737602 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:25.125968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:25.125940 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x"] Nov 03 22:42:25.126396 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:42:25.126279 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4fcf00_89e4_45e8_95b1_0e272580b7ca.slice/crio-77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67 WatchSource:0}: Error finding container 77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67: Status 404 returned error can't find the container with id 77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67 Nov 03 22:42:25.178611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:25.178580 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" event={"ID":"ce4fcf00-89e4-45e8-95b1-0e272580b7ca","Type":"ContainerStarted","Data":"77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67"} Nov 03 22:42:26.187281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:26.187245 5051 generic.go:356] "Generic (PLEG): container finished" podID="ce4fcf00-89e4-45e8-95b1-0e272580b7ca" containerID="04f3463ec13b92360f732e2cf072384ba7c5666905e91a08619195b569ef8d4b" exitCode=0 Nov 03 22:42:26.187689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:26.187326 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" event={"ID":"ce4fcf00-89e4-45e8-95b1-0e272580b7ca","Type":"ContainerDied","Data":"04f3463ec13b92360f732e2cf072384ba7c5666905e91a08619195b569ef8d4b"} Nov 03 22:42:27.364791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.364768 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:27.543819 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.543745 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\") pod \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " Nov 03 22:42:27.543959 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.543882 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\" (UniqueName: \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\") pod \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " Nov 03 22:42:27.543959 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.543936 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\") pod \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\" (UID: \"ce4fcf00-89e4-45e8-95b1-0e272580b7ca\") " Nov 03 22:42:27.544199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.544172 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca" pod "ce4fcf00-89e4-45e8-95b1-0e272580b7ca" (UID: "ce4fcf00-89e4-45e8-95b1-0e272580b7ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:27.544626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.544606 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:27.546868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.546839 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca" pod "ce4fcf00-89e4-45e8-95b1-0e272580b7ca" (UID: "ce4fcf00-89e4-45e8-95b1-0e272580b7ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:27.546971 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.546847 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm" pod "ce4fcf00-89e4-45e8-95b1-0e272580b7ca" (UID: "ce4fcf00-89e4-45e8-95b1-0e272580b7ca"). InnerVolumeSpecName "kube-api-access-gfbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:27.646228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.646200 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbbm\" (UniqueName: \"kubernetes.io/projected/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-kube-api-access-gfbbm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:27.646228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:27.646225 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4fcf00-89e4-45e8-95b1-0e272580b7ca-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:28.203508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:28.203474 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" Nov 03 22:42:28.203695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:28.203476 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x" event={"ID":"ce4fcf00-89e4-45e8-95b1-0e272580b7ca","Type":"ContainerDied","Data":"77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67"} Nov 03 22:42:28.203695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:28.203602 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b5a6de430be7a4707a0477ff080f8a70f8e00118d02e1388428c9cdcf29c67" Nov 03 22:42:34.863056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.863006 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9"] Nov 03 22:42:34.863687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.863670 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ce4fcf00-89e4-45e8-95b1-0e272580b7ca" containerName="revoke-keys" Nov 03 22:42:34.863687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.863686 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4fcf00-89e4-45e8-95b1-0e272580b7ca" containerName="revoke-keys" Nov 03 22:42:34.863819 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.863809 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4fcf00-89e4-45e8-95b1-0e272580b7ca" containerName="revoke-keys" Nov 03 22:42:34.867883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.867858 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:34.925162 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.925126 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9"] Nov 03 22:42:34.943749 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.943705 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:34.943964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.943823 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:34.943964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.943919 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsf2\" (UniqueName: \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:34.968493 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.968446 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:34.968691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:34.968515 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:42:35.049105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.049059 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.049331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.049136 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.049331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.049214 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsf2\" (UniqueName: \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.050517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.050485 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.062049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.061953 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2"] Nov 03 22:42:35.062611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.062558 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.065782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.065742 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee382c38-k75w2"] Nov 03 22:42:35.110605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.110571 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsf2\" (UniqueName: \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\") pod \"maas-api-revoke-sub-56bcb65e-rgww9\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.182759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.182664 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:35.566879 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:42:35.566837 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e6a0a0_b6df_4080_a98c_baad0b934992.slice/crio-38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286 WatchSource:0}: Error finding container 38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286: Status 404 returned error can't find the container with id 38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286 Nov 03 22:42:35.567067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:35.566927 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9"] Nov 03 22:42:36.288690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:36.288599 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" event={"ID":"a3e6a0a0-b6df-4080-a98c-baad0b934992","Type":"ContainerStarted","Data":"2418fca28a464e2194461f67f14486bf3c3ca5f0f53949e6f445bba2077d13ba"} Nov 03 22:42:36.288690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:36.288664 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" event={"ID":"a3e6a0a0-b6df-4080-a98c-baad0b934992","Type":"ContainerStarted","Data":"38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286"} Nov 03 22:42:36.440352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:36.440314 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9563d519-9066-4803-8974-cc58ef958560" path="/var/lib/kubelet/pods/9563d519-9066-4803-8974-cc58ef958560/volumes" Nov 03 22:42:37.298730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:37.298695 5051 generic.go:356] "Generic (PLEG): container finished" podID="a3e6a0a0-b6df-4080-a98c-baad0b934992" containerID="2418fca28a464e2194461f67f14486bf3c3ca5f0f53949e6f445bba2077d13ba" exitCode=0 Nov 03 22:42:37.299291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:37.298787 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" event={"ID":"a3e6a0a0-b6df-4080-a98c-baad0b934992","Type":"ContainerDied","Data":"2418fca28a464e2194461f67f14486bf3c3ca5f0f53949e6f445bba2077d13ba"} Nov 03 22:42:38.477180 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.477152 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:38.609935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.609898 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\") pod \"a3e6a0a0-b6df-4080-a98c-baad0b934992\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " Nov 03 22:42:38.610147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.609985 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\" (UniqueName: \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\") pod \"a3e6a0a0-b6df-4080-a98c-baad0b934992\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " Nov 03 22:42:38.610147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.610136 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\") pod \"a3e6a0a0-b6df-4080-a98c-baad0b934992\" (UID: \"a3e6a0a0-b6df-4080-a98c-baad0b934992\") " Nov 03 22:42:38.610825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.610611 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca" pod "a3e6a0a0-b6df-4080-a98c-baad0b934992" (UID: "a3e6a0a0-b6df-4080-a98c-baad0b934992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:38.610825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.610800 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.612372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.612348 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2" pod "a3e6a0a0-b6df-4080-a98c-baad0b934992" (UID: "a3e6a0a0-b6df-4080-a98c-baad0b934992"). InnerVolumeSpecName "kube-api-access-zjsf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:38.627278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.627255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca" pod "a3e6a0a0-b6df-4080-a98c-baad0b934992" (UID: "a3e6a0a0-b6df-4080-a98c-baad0b934992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:38.711626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.711593 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjsf2\" (UniqueName: \"kubernetes.io/projected/a3e6a0a0-b6df-4080-a98c-baad0b934992-kube-api-access-zjsf2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.711626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:38.711619 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e6a0a0-b6df-4080-a98c-baad0b934992-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:39.320789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:39.320761 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" Nov 03 22:42:39.320964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:39.320931 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9" event={"ID":"a3e6a0a0-b6df-4080-a98c-baad0b934992","Type":"ContainerDied","Data":"38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286"} Nov 03 22:42:39.321021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:39.320971 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38170daad8bddf289d78f6009d3fc0aec4a465896c3b6cdbc7ad7d87bc040286" Nov 03 22:42:43.215408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.215366 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 22:42:43.216003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.215975 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a3e6a0a0-b6df-4080-a98c-baad0b934992" containerName="revoke-keys" Nov 03 22:42:43.216063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.216004 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e6a0a0-b6df-4080-a98c-baad0b934992" containerName="revoke-keys" Nov 03 22:42:43.216132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.216123 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e6a0a0-b6df-4080-a98c-baad0b934992" containerName="revoke-keys" Nov 03 22:42:43.220071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.220049 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.277098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.277067 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 22:42:43.310681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.310654 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.310829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.310784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.310883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.310865 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.310927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.310913 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffv9m\" (UniqueName: \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.310967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.310956 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.311092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.311071 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.321356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.321333 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7540da\"/\"distinct-2-w6-7540da-kserve-self-signed-certs\"" Nov 03 22:42:43.413240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413204 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.413440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413338 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.413440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413371 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffv9m\" (UniqueName: \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.413562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.413562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413518 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.413666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.413584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.415011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.414095 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.415011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.414696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.415216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.415092 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.419128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.419107 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.419368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.419347 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.477774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.477691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffv9m\" (UniqueName: \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\") pod \"distinct-2-w6-7540da-kserve-757c67b5ff-z4sml\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.532828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.532795 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:43.962253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:43.962226 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 22:42:43.963241 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:42:43.963215 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode434e52e_2636_4146_a822_ce3deee299bd.slice/crio-ad06d00d3eb97807414c4835515833a743a771c754daeb2dab8fa00a97af0769 WatchSource:0}: Error finding container ad06d00d3eb97807414c4835515833a743a771c754daeb2dab8fa00a97af0769: Status 404 returned error can't find the container with id ad06d00d3eb97807414c4835515833a743a771c754daeb2dab8fa00a97af0769 Nov 03 22:42:44.372040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:44.372001 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" event={"ID":"e434e52e-2636-4146-a822-ce3deee299bd","Type":"ContainerStarted","Data":"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117"} Nov 03 22:42:44.372506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:44.372055 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" event={"ID":"e434e52e-2636-4146-a822-ce3deee299bd","Type":"ContainerStarted","Data":"ad06d00d3eb97807414c4835515833a743a771c754daeb2dab8fa00a97af0769"} Nov 03 22:42:44.494796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:44.494728 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podStartSLOduration=1.49470872 podStartE2EDuration="1.49470872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:44.487606458 +0000 UTC m=+1238.224950684" watchObservedRunningTime="2026-11-03 22:42:44.49470872 +0000 UTC m=+1238.232052946" Nov 03 22:42:44.533158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:44.533126 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:51.048829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:51.048792 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn"] Nov 03 22:42:51.098355 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:51.098326 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d39c3d-f4nkn"] Nov 03 22:42:51.622520 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:42:51.622485 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:42:52.410061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:52.410032 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a40e3c-d61c-4346-b9e4-4c00fcb773c9" path="/var/lib/kubelet/pods/33a40e3c-d61c-4346-b9e4-4c00fcb773c9/volumes" Nov 03 22:42:53.021094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.021055 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk"] Nov 03 22:42:53.025666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.025644 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.045508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.045465 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.045722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.045709 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.045781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.045730 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2ss\" (UniqueName: \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.082968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.082931 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk"] Nov 03 22:42:53.126274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.126238 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:42:53.126445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.126324 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:42:53.147067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.147031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.147240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.147080 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2ss\" (UniqueName: \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.147240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.147156 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.148313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.148269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.158939 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.158907 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.209662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.209635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2ss\" (UniqueName: \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\") pod \"maas-api-revoke-sub-f67bd5d5-qtddk\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.341005 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.340912 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:53.533661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.533614 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:53.547072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.547027 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:53.724787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:53.724744 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk"] Nov 03 22:42:53.727162 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:42:53.727129 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9a2ff2_95f5_47e4_9941_e2f09ac011f0.slice/crio-02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f WatchSource:0}: Error finding container 02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f: Status 404 returned error can't find the container with id 02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f Nov 03 22:42:54.463361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:54.463320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" event={"ID":"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0","Type":"ContainerStarted","Data":"b10c00e0e00cf50590491ef58240386d73186abd130487b8e6774bac0ca1ff98"} Nov 03 22:42:54.463594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:54.463380 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" event={"ID":"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0","Type":"ContainerStarted","Data":"02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f"} Nov 03 22:42:54.475455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:54.475423 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:42:54.582595 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:54.582535 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" podStartSLOduration=2.582516101 podStartE2EDuration="2.582516101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:54.577195603 +0000 UTC m=+1248.314539830" watchObservedRunningTime="2026-11-03 22:42:54.582516101 +0000 UTC m=+1248.319860325" Nov 03 22:42:55.473317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:55.473284 5051 generic.go:356] "Generic (PLEG): container finished" podID="dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" containerID="b10c00e0e00cf50590491ef58240386d73186abd130487b8e6774bac0ca1ff98" exitCode=0 Nov 03 22:42:55.473602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:55.473427 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" event={"ID":"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0","Type":"ContainerDied","Data":"b10c00e0e00cf50590491ef58240386d73186abd130487b8e6774bac0ca1ff98"} Nov 03 22:42:56.640117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.640091 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:56.696189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.696162 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\") pod \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " Nov 03 22:42:56.696345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.696332 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\") pod \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " Nov 03 22:42:56.696387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.696364 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\" (UniqueName: \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\") pod \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\" (UID: \"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0\") " Nov 03 22:42:56.696660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.696628 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca" pod "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" (UID: "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:56.696875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.696857 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:56.698394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.698368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca" pod "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" (UID: "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:56.698536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.698516 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss" pod "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" (UID: "dd9a2ff2-95f5-47e4-9941-e2f09ac011f0"). InnerVolumeSpecName "kube-api-access-4c2ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:56.798576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.798493 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:56.798576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:56.798536 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2ss\" (UniqueName: \"kubernetes.io/projected/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0-kube-api-access-4c2ss\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:42:57.492306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:57.492271 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" Nov 03 22:42:57.492306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:57.492297 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk" event={"ID":"dd9a2ff2-95f5-47e4-9941-e2f09ac011f0","Type":"ContainerDied","Data":"02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f"} Nov 03 22:42:57.492576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:42:57.492341 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02330b8cf0128da1755ca9ff1a27a79778a8abab50d3e5d099b54f43832ca12f" Nov 03 22:43:02.050528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:02.050494 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4"] Nov 03 22:43:02.098745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:02.098713 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff96fe4c-257f4"] Nov 03 22:43:02.411113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:02.411082 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a077ff-dc68-4334-afa9-8fc5d25e30c1" path="/var/lib/kubelet/pods/62a077ff-dc68-4334-afa9-8fc5d25e30c1/volumes" Nov 03 22:43:03.026910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.026868 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:43:03.027644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.027615 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" containerName="revoke-keys" Nov 03 22:43:03.027644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.027644 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" containerName="revoke-keys" Nov 03 22:43:03.027852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.027838 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" containerName="revoke-keys" Nov 03 22:43:03.032246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.032217 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.076346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.076295 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:43:03.081030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.080971 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.133718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.133683 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:43:03.184958 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.184918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24grq\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.184984 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.185206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185046 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185135 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.185388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185260 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.185388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185322 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185398 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185452 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.185501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185485 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185540 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.185653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185632 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.185865 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185836 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.186018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185879 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.186018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.185950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.186149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.186035 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.186205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.186137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.186205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.186198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.186305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.186235 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr8l\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.188567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.188542 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288055 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288148 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288175 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288247 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288299 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288369 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288438 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288484 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288571 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288742 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288769 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288799 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288824 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288862 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288893 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288922 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr8l\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.288963 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24grq\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.289776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.289492 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.293064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.291945 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.293064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.292237 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.294305 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.294558 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.294781 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.295394 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.298104 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.298830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.298788 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.299263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.299228 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.299678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.299641 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.303832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.303283 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.303832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.303785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.309393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.309362 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.352285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.352242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24grq\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.355238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.355207 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.356118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.356084 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.357930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.357901 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr8l\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.400186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.400144 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:03.446354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.446309 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:43:03.453299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.453269 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.520029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.518694 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:43:03.598494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598389 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598541 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598634 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598712 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598771 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598808 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.598887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.598871 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrvp\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.599056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.599035 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.599119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.599080 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.647770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.647733 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:03.701035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.700958 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701050 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701251 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701314 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701381 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701418 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701448 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701491 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701675 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrvp\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701828 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.701912 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.701849 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.702183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.702159 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.702646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.702619 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.702935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.702666 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.705292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.705257 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.705603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.705577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.768348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.768313 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.770348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.770250 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrvp\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.779175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.779140 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:03.896956 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.896924 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:43:03.899417 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:03.899383 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642079d7_6d23_43e7_8625_32da34f6ca3a.slice/crio-11a2586210836c565849ba29bfd5054aabd5b23d3f525782aaea68c687939d90 WatchSource:0}: Error finding container 11a2586210836c565849ba29bfd5054aabd5b23d3f525782aaea68c687939d90: Status 404 returned error can't find the container with id 11a2586210836c565849ba29bfd5054aabd5b23d3f525782aaea68c687939d90 Nov 03 22:43:03.903400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.903356 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:03.903504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.903446 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:03.903504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:03.903495 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.048651 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:04.048622 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8669183_3138_4255_b575_24abbd91ba84.slice/crio-f1896091c07be613ebbedcdd07945e3a71590295d022662edc084ba013a4f82e WatchSource:0}: Error finding container f1896091c07be613ebbedcdd07945e3a71590295d022662edc084ba013a4f82e: Status 404 returned error can't find the container with id f1896091c07be613ebbedcdd07945e3a71590295d022662edc084ba013a4f82e Nov 03 22:43:04.048972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.048950 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:43:04.051872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.051833 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.052052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.051932 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.052052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.051975 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.199059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.199022 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:43:04.199478 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:04.199083 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa83d4ea_bb36_4dd0_872b_a877e824f2b2.slice/crio-a35616ffd145b127643343bf5f6d1e21ef5d000f2fb6c5c08129dc92bd3add13 WatchSource:0}: Error finding container a35616ffd145b127643343bf5f6d1e21ef5d000f2fb6c5c08129dc92bd3add13: Status 404 returned error can't find the container with id a35616ffd145b127643343bf5f6d1e21ef5d000f2fb6c5c08129dc92bd3add13 Nov 03 22:43:04.203409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.203370 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.203516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.203455 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.203516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.203496 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.557418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.557324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" event={"ID":"642079d7-6d23-43e7-8625-32da34f6ca3a","Type":"ContainerStarted","Data":"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c"} Nov 03 22:43:04.557418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.557384 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" event={"ID":"642079d7-6d23-43e7-8625-32da34f6ca3a","Type":"ContainerStarted","Data":"11a2586210836c565849ba29bfd5054aabd5b23d3f525782aaea68c687939d90"} Nov 03 22:43:04.561037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.561011 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" event={"ID":"c8669183-3138-4255-b575-24abbd91ba84","Type":"ContainerStarted","Data":"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf"} Nov 03 22:43:04.561246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.561061 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" event={"ID":"c8669183-3138-4255-b575-24abbd91ba84","Type":"ContainerStarted","Data":"f1896091c07be613ebbedcdd07945e3a71590295d022662edc084ba013a4f82e"} Nov 03 22:43:04.562825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.562792 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" event={"ID":"fa83d4ea-bb36-4dd0-872b-a877e824f2b2","Type":"ContainerStarted","Data":"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3"} Nov 03 22:43:04.562939 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.562840 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" event={"ID":"fa83d4ea-bb36-4dd0-872b-a877e824f2b2","Type":"ContainerStarted","Data":"a35616ffd145b127643343bf5f6d1e21ef5d000f2fb6c5c08129dc92bd3add13"} Nov 03 22:43:04.648781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.648591 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:04.692949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.692889 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podStartSLOduration=1.6928693799999999 podStartE2EDuration="1.69286938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:04.6913435 +0000 UTC m=+1258.428687740" watchObservedRunningTime="2026-11-03 22:43:04.69286938 +0000 UTC m=+1258.430213607" Nov 03 22:43:04.780011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.779946 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:04.811832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.811694 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podStartSLOduration=2.81167401 podStartE2EDuration="2.81167401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:04.810022982 +0000 UTC m=+1258.547367208" watchObservedRunningTime="2026-11-03 22:43:04.81167401 +0000 UTC m=+1258.549018237" Nov 03 22:43:04.937392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:04.937340 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podStartSLOduration=1.937323938 podStartE2EDuration="1.937323938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:04.930313375 +0000 UTC m=+1258.667657601" watchObservedRunningTime="2026-11-03 22:43:04.937323938 +0000 UTC m=+1258.674668163" Nov 03 22:43:05.404566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.403113 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:05.404566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.403247 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.404566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.403293 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:05.671069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.668138 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.671069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.668196 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:05.800012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.797545 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.800012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:05.797597 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:06.427211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.425867 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.427211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.425932 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:06.652215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.652181 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.652479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.652452 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:06.750081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.749322 5051 scope.go:122] "RemoveContainer" containerID="bce408617c31a07626f435588af23bd93cafe1995c33ca4e126016c4b7bb596e" Nov 03 22:43:06.789029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.783836 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.789029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:06.783888 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:07.157010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.153146 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv"] Nov 03 22:43:07.232015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.231262 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e501ba38-7swkv"] Nov 03 22:43:07.424017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.423141 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.424017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.423208 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:07.649009 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.648163 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.649009 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.648223 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:07.793071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.792455 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.793071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:07.792516 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:08.408017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.407438 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.408017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.407502 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:08.438018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.431516 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82957650-f195-4666-a95d-228727c46d47" path="/var/lib/kubelet/pods/82957650-f195-4666-a95d-228727c46d47/volumes" Nov 03 22:43:08.648871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.648836 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.649204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.649157 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:08.781367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.781270 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.781659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:08.781611 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:09.242071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.208268 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7"] Nov 03 22:43:09.346065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.345052 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7"] Nov 03 22:43:09.346065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.345186 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.408692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.408659 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:43:09.410839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.410812 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.426012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.416438 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.435903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.419202 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.471201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.471137 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:09.537077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.501462 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:09.575262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.562191 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4klj\" (UniqueName: \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.679136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.675533 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.679136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.675593 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:09.681706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.681674 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.682930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.682907 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.683212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.683192 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4klj\" (UniqueName: \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.694012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.684471 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.735693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.735651 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.769652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.765308 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4klj\" (UniqueName: \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\") pod \"maas-api-revoke-sub-04e66e53-gf4c7\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.804141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.803740 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.804331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.804257 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:09.806475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.806445 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.806653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.806630 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:09.817109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.817073 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.857591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.834248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:43:09.940574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.940537 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.941159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.940812 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:09.985197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:09.985099 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.005532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.005492 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:43:10.239252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.239204 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.257604 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.257459 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.309772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.309717 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:43:10.310534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.310486 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.384014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.362279 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.455108 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.455061 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.466170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.466085 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:10.509109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.480172 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.642041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.613074 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.656020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.652365 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.660013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.656265 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:10.796113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.781907 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.796113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.781960 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:10.809343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.809261 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:43:10.817181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.817132 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:43:10.825482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.825402 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.867053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.842350 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:43:10.905406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.905324 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.985066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.933099 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:43:10.985066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.960147 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.985066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.962020 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:43:10.985066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.963392 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:10.985066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:10.980327 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:43:11.011933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.010911 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.013105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.012189 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.017025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.013341 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:43:11.017232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.017214 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.285211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.294108 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.336075 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.336566 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.337515 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.337748 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.420323 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.420803 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.447232 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.569650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.447289 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:11.663142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.582725 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:43:11.663142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.641509 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:43:11.809127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.809072 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.906016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.886008 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:43:11.935433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.935354 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:43:11.936801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.936762 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.984382 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.984335 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.985945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:11.985910 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 22:43:12.034149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.032785 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7"] Nov 03 22:43:12.056098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.056063 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:43:12.073385 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.073362 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:43:12.073856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.073828 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 03 22:43:12.094406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.094368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:43:12.096016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.095965 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.314070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.313580 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.314070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.313645 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:12.314070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.313747 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.314070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.313791 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:12.433420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.433382 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:43:12.434100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.434064 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:43:12.580181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.493217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 03 22:43:12.611119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.611072 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.666128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.666090 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.973422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.973380 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:12.973422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.869268 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.974021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.822591 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r7qkw" podUID="69267d17-a659-4d2d-ae3b-2560c2968dec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.974021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.848960 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.974021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.973908 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:12.974021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.939863 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:43:12.974269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.924925 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.974269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.934534 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" event={"ID":"2978bc99-df8f-41b3-9e03-59f7196c8de3","Type":"ContainerStarted","Data":"7ca482205ccc3ba503c1cfd49efa185ce938764d85f7846011a1893304490599"} Nov 03 22:43:12.974391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.932581 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.974391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.974349 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:12.974391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.939835 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:12.974717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.973146 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:12.974717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:12.974585 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.095682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.095643 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.096696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.096653 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.162097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.156089 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:43:13.162097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.156150 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:43:13.162097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.156185 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:43:13.162097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.156213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:43:13.440903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.439360 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:13.516352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.516321 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:43:13.547029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.539863 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:43:13.583054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.582832 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.583054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.582887 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.593318 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.594121 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.595376 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.595411 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.595468 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:43:13.605024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.595520 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:43:13.648272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.648239 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.669103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.669067 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 22:43:13.781349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.781315 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 22:43:13.781502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:13.781375 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 22:43:14.530833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.530803 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:14.532632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.532230 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:14.535937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.535890 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:43:14.780495 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.779093 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:14.780495 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.780323 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:14.781681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.781608 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:43:14.885465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.885433 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:14.886703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.886630 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:14.888019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.887984 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:43:14.966734 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.966702 5051 generic.go:356] "Generic (PLEG): container finished" podID="2978bc99-df8f-41b3-9e03-59f7196c8de3" containerID="1792d4065fb7ea7e8b5b492da22e3de48da13e017d4bb2154b6e4567e938b67d" exitCode=0 Nov 03 22:43:14.967451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:14.967088 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" event={"ID":"2978bc99-df8f-41b3-9e03-59f7196c8de3","Type":"ContainerDied","Data":"1792d4065fb7ea7e8b5b492da22e3de48da13e017d4bb2154b6e4567e938b67d"} Nov 03 22:43:16.246029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.245942 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq"] Nov 03 22:43:16.249296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.249276 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:16.250325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.250310 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.309686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.309657 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq"] Nov 03 22:43:16.320302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.320275 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\") pod \"2978bc99-df8f-41b3-9e03-59f7196c8de3\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " Nov 03 22:43:16.320466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.320430 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\") pod \"2978bc99-df8f-41b3-9e03-59f7196c8de3\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " Nov 03 22:43:16.320531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.320512 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\" (UniqueName: \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\") pod \"2978bc99-df8f-41b3-9e03-59f7196c8de3\" (UID: \"2978bc99-df8f-41b3-9e03-59f7196c8de3\") " Nov 03 22:43:16.320830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.320804 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.321157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.321068 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.321279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.321220 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntgj\" (UniqueName: \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.321279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.321224 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca" pod "2978bc99-df8f-41b3-9e03-59f7196c8de3" (UID: "2978bc99-df8f-41b3-9e03-59f7196c8de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:16.321453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.321430 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:16.322700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.322678 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca" pod "2978bc99-df8f-41b3-9e03-59f7196c8de3" (UID: "2978bc99-df8f-41b3-9e03-59f7196c8de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:16.323930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.323906 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj" pod "2978bc99-df8f-41b3-9e03-59f7196c8de3" (UID: "2978bc99-df8f-41b3-9e03-59f7196c8de3"). InnerVolumeSpecName "kube-api-access-j4klj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.422492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntgj\" (UniqueName: \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.422625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.422782 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.423026 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2978bc99-df8f-41b3-9e03-59f7196c8de3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.423102 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4klj\" (UniqueName: \"kubernetes.io/projected/2978bc99-df8f-41b3-9e03-59f7196c8de3-kube-api-access-j4klj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.423696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.426036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.425617 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.486403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.486373 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntgj\" (UniqueName: \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\") pod \"maas-api-revoke-sub-1012649d-zndxq\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.567926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.567851 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:16.957880 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:16.957852 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5147edd_f9cf_4942_b254_d9544100c207.slice/crio-b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b WatchSource:0}: Error finding container b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b: Status 404 returned error can't find the container with id b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b Nov 03 22:43:16.958570 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.958551 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq"] Nov 03 22:43:16.998790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.998769 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" Nov 03 22:43:16.998898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.998810 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7" event={"ID":"2978bc99-df8f-41b3-9e03-59f7196c8de3","Type":"ContainerDied","Data":"7ca482205ccc3ba503c1cfd49efa185ce938764d85f7846011a1893304490599"} Nov 03 22:43:16.998898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:16.998867 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca482205ccc3ba503c1cfd49efa185ce938764d85f7846011a1893304490599" Nov 03 22:43:17.000885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:17.000864 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" event={"ID":"b5147edd-f9cf-4942-b254-d9544100c207","Type":"ContainerStarted","Data":"b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b"} Nov 03 22:43:18.010614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:18.010582 5051 generic.go:356] "Generic (PLEG): container finished" podID="b5147edd-f9cf-4942-b254-d9544100c207" containerID="a6cbdbcf0b3b4fd3a421a89366ad0e739d2afd6748b19a3968b416280f500df3" exitCode=0 Nov 03 22:43:18.010955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:18.010658 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" event={"ID":"b5147edd-f9cf-4942-b254-d9544100c207","Type":"ContainerDied","Data":"a6cbdbcf0b3b4fd3a421a89366ad0e739d2afd6748b19a3968b416280f500df3"} Nov 03 22:43:19.180071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.180050 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:19.260555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.260530 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\") pod \"b5147edd-f9cf-4942-b254-d9544100c207\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " Nov 03 22:43:19.260757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.260740 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\" (UniqueName: \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\") pod \"b5147edd-f9cf-4942-b254-d9544100c207\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " Nov 03 22:43:19.260919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.260887 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\") pod \"b5147edd-f9cf-4942-b254-d9544100c207\" (UID: \"b5147edd-f9cf-4942-b254-d9544100c207\") " Nov 03 22:43:19.261972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.261941 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca" pod "b5147edd-f9cf-4942-b254-d9544100c207" (UID: "b5147edd-f9cf-4942-b254-d9544100c207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:19.263034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.263004 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj" pod "b5147edd-f9cf-4942-b254-d9544100c207" (UID: "b5147edd-f9cf-4942-b254-d9544100c207"). InnerVolumeSpecName "kube-api-access-qntgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:19.263141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.263100 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca" pod "b5147edd-f9cf-4942-b254-d9544100c207" (UID: "b5147edd-f9cf-4942-b254-d9544100c207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:19.352953 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.352913 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw"] Nov 03 22:43:19.354046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354023 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b5147edd-f9cf-4942-b254-d9544100c207" containerName="revoke-keys" Nov 03 22:43:19.354216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354095 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5147edd-f9cf-4942-b254-d9544100c207" containerName="revoke-keys" Nov 03 22:43:19.354216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354142 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2978bc99-df8f-41b3-9e03-59f7196c8de3" containerName="revoke-keys" Nov 03 22:43:19.354216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354151 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2978bc99-df8f-41b3-9e03-59f7196c8de3" containerName="revoke-keys" Nov 03 22:43:19.354380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354356 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2978bc99-df8f-41b3-9e03-59f7196c8de3" containerName="revoke-keys" Nov 03 22:43:19.354430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.354381 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5147edd-f9cf-4942-b254-d9544100c207" containerName="revoke-keys" Nov 03 22:43:19.358608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.358515 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.363928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.363898 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:19.364083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.363934 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5147edd-f9cf-4942-b254-d9544100c207-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:19.364083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.363949 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntgj\" (UniqueName: \"kubernetes.io/projected/b5147edd-f9cf-4942-b254-d9544100c207-kube-api-access-qntgj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:19.378884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.377513 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw"] Nov 03 22:43:19.465799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.465766 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.465948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.465813 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.465948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.465929 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7sb\" (UniqueName: \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.567912 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.567823 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.567912 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.567868 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.568174 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.567915 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7sb\" (UniqueName: \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.568719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.568691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.570450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.570425 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.628594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.628573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7sb\" (UniqueName: \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\") pod \"maas-api-revoke-sub-8a106282-zc9bw\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:19.685835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:19.685804 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:20.028646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.028621 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" Nov 03 22:43:20.028815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.028614 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq" event={"ID":"b5147edd-f9cf-4942-b254-d9544100c207","Type":"ContainerDied","Data":"b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b"} Nov 03 22:43:20.028815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.028741 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89e042e9a1a90ad3ce582f55bdef13e7861107c74bf673548190c5bfc18457b" Nov 03 22:43:20.094267 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:20.094239 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec767b9_d2fd_481d_9824_46ab3fb2b290.slice/crio-6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12 WatchSource:0}: Error finding container 6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12: Status 404 returned error can't find the container with id 6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12 Nov 03 22:43:20.096921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.096883 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw"] Nov 03 22:43:20.181008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.180081 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:43:20.194567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.194539 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.241513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.241476 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:43:20.282668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.282514 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.282668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.282558 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksf58\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.282668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.282588 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.282916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.282793 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.282916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.282835 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.283073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.283012 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.283201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.283185 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.283284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.283226 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.283339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.283317 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385101 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385140 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksf58\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385166 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385211 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385236 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385326 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385446 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385491 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.385693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385659 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.386227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.385804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.386227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.386052 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.386227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.386093 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.386546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.386520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.387802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.387777 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.387958 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.387939 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.445834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.445809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksf58\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.445938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.445813 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.511084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.511058 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:20.953183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.953156 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:43:20.953643 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:20.953620 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37830e06_896a_49da_88d9_a8eadc034556.slice/crio-a8b950b6cd19963ca2e1a16a273bd91acfcb431326a195158b0ba33c060ca827 WatchSource:0}: Error finding container a8b950b6cd19963ca2e1a16a273bd91acfcb431326a195158b0ba33c060ca827: Status 404 returned error can't find the container with id a8b950b6cd19963ca2e1a16a273bd91acfcb431326a195158b0ba33c060ca827 Nov 03 22:43:20.956198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.956163 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:20.956306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.956246 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:20.956306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:20.956275 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:21.039343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.039311 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" event={"ID":"5ec767b9-d2fd-481d-9824-46ab3fb2b290","Type":"ContainerStarted","Data":"482a22410cf4f2037394908b475bbc62514720946b7eed4b0f82aaf583857fd6"} Nov 03 22:43:21.039482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.039372 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" event={"ID":"5ec767b9-d2fd-481d-9824-46ab3fb2b290","Type":"ContainerStarted","Data":"6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12"} Nov 03 22:43:21.041458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.041430 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" event={"ID":"37830e06-896a-49da-88d9-a8eadc034556","Type":"ContainerStarted","Data":"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250"} Nov 03 22:43:21.041575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.041490 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" event={"ID":"37830e06-896a-49da-88d9-a8eadc034556","Type":"ContainerStarted","Data":"a8b950b6cd19963ca2e1a16a273bd91acfcb431326a195158b0ba33c060ca827"} Nov 03 22:43:21.159418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.159336 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" podStartSLOduration=2.159317958 podStartE2EDuration="2.159317958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:21.15714466 +0000 UTC m=+1274.894488885" watchObservedRunningTime="2026-11-03 22:43:21.159317958 +0000 UTC m=+1274.896662184" Nov 03 22:43:21.282591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.282479 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podStartSLOduration=1.2824605519999999 podStartE2EDuration="1.282460552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:21.275558578 +0000 UTC m=+1275.012902807" watchObservedRunningTime="2026-11-03 22:43:21.282460552 +0000 UTC m=+1275.019804779" Nov 03 22:43:21.511807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:21.511775 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:21.624359 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:43:21.624324 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:43:22.050910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.050832 5051 generic.go:356] "Generic (PLEG): container finished" podID="5ec767b9-d2fd-481d-9824-46ab3fb2b290" containerID="482a22410cf4f2037394908b475bbc62514720946b7eed4b0f82aaf583857fd6" exitCode=0 Nov 03 22:43:22.051072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.050926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" event={"ID":"5ec767b9-d2fd-481d-9824-46ab3fb2b290","Type":"ContainerDied","Data":"482a22410cf4f2037394908b475bbc62514720946b7eed4b0f82aaf583857fd6"} Nov 03 22:43:22.051683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.051668 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk"] Nov 03 22:43:22.072013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.071198 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c09441-sjlfk"] Nov 03 22:43:22.409067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.409037 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb02a71-e102-46af-80d8-bc02f2e7454a" path="/var/lib/kubelet/pods/1eb02a71-e102-46af-80d8-bc02f2e7454a/volumes" Nov 03 22:43:22.513462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.513425 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:22.513661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:22.513477 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:23.287494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.287470 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:23.341391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.341321 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\" (UniqueName: \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\") pod \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " Nov 03 22:43:23.341537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.341400 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\") pod \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " Nov 03 22:43:23.341537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.341460 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\") pod \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\" (UID: \"5ec767b9-d2fd-481d-9824-46ab3fb2b290\") " Nov 03 22:43:23.341781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.341758 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca" pod "5ec767b9-d2fd-481d-9824-46ab3fb2b290" (UID: "5ec767b9-d2fd-481d-9824-46ab3fb2b290"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:23.342060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.342042 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:23.343618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.343592 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb" pod "5ec767b9-d2fd-481d-9824-46ab3fb2b290" (UID: "5ec767b9-d2fd-481d-9824-46ab3fb2b290"). InnerVolumeSpecName "kube-api-access-2r7sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:23.343732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.343718 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca" pod "5ec767b9-d2fd-481d-9824-46ab3fb2b290" (UID: "5ec767b9-d2fd-481d-9824-46ab3fb2b290"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:23.443591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.443560 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7sb\" (UniqueName: \"kubernetes.io/projected/5ec767b9-d2fd-481d-9824-46ab3fb2b290-kube-api-access-2r7sb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:23.443591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.443587 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec767b9-d2fd-481d-9824-46ab3fb2b290-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:23.511387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.511359 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:23.511553 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:23.511405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:24.091070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:24.091022 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" event={"ID":"5ec767b9-d2fd-481d-9824-46ab3fb2b290","Type":"ContainerDied","Data":"6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12"} Nov 03 22:43:24.091275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:24.091094 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae2614628cf010daa50e1ff390c74a33a9bff588fb91af5e70f2dc6e2346e12" Nov 03 22:43:24.091275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:24.091043 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw" Nov 03 22:43:24.512420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:24.512390 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:24.512820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:24.512442 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:25.511697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:25.511666 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:25.511878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:25.511717 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:26.512342 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:26.512313 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:26.512836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:26.512372 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:27.511461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:27.511421 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:43:27.511773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:27.511472 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:43:28.516451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:28.516414 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:28.516892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:28.516869 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:28.517677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:28.517657 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:43:31.612300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.612261 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx"] Nov 03 22:43:31.613824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.613796 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5ec767b9-d2fd-481d-9824-46ab3fb2b290" containerName="revoke-keys" Nov 03 22:43:31.613824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.613824 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec767b9-d2fd-481d-9824-46ab3fb2b290" containerName="revoke-keys" Nov 03 22:43:31.614201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.614183 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec767b9-d2fd-481d-9824-46ab3fb2b290" containerName="revoke-keys" Nov 03 22:43:31.619867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.619840 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.671600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.671567 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx"] Nov 03 22:43:31.687590 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.687563 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.687798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.687627 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.687798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.687776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrkv\" (UniqueName: \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.714556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.714523 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:31.714757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.714665 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:43:31.789081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.789048 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.789270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.789138 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrkv\" (UniqueName: \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.789335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.789311 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.789723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.789702 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.791766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.791746 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.849982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.849948 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrkv\" (UniqueName: \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\") pod \"maas-api-revoke-sub-31659a80-4j5zx\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:31.937523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:31.937433 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:32.331705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:32.331675 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx"] Nov 03 22:43:32.331945 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:32.331920 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbf2e61_1345_4f5e_9710_4bddac89800a.slice/crio-1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735 WatchSource:0}: Error finding container 1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735: Status 404 returned error can't find the container with id 1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735 Nov 03 22:43:33.056958 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.056884 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k"] Nov 03 22:43:33.108244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.108207 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj"] Nov 03 22:43:33.112852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.112826 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59672595-72f6k"] Nov 03 22:43:33.116962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.116944 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5030940-94nmj"] Nov 03 22:43:33.175237 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.175208 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" event={"ID":"6fbf2e61-1345-4f5e-9710-4bddac89800a","Type":"ContainerStarted","Data":"c07514cf8e3b6fe7ba54ecaf3f522aaac87009cbf0de0cc75a5ea02168ded3a0"} Nov 03 22:43:33.175368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.175265 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" event={"ID":"6fbf2e61-1345-4f5e-9710-4bddac89800a","Type":"ContainerStarted","Data":"1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735"} Nov 03 22:43:33.297860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:33.297812 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" podStartSLOduration=2.297797929 podStartE2EDuration="2.297797929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:33.29023704 +0000 UTC m=+1287.027581266" watchObservedRunningTime="2026-11-03 22:43:33.297797929 +0000 UTC m=+1287.035142159" Nov 03 22:43:34.185793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:34.185764 5051 generic.go:356] "Generic (PLEG): container finished" podID="6fbf2e61-1345-4f5e-9710-4bddac89800a" containerID="c07514cf8e3b6fe7ba54ecaf3f522aaac87009cbf0de0cc75a5ea02168ded3a0" exitCode=0 Nov 03 22:43:34.186223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:34.185840 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" event={"ID":"6fbf2e61-1345-4f5e-9710-4bddac89800a","Type":"ContainerDied","Data":"c07514cf8e3b6fe7ba54ecaf3f522aaac87009cbf0de0cc75a5ea02168ded3a0"} Nov 03 22:43:34.411596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:34.411559 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1392b7-3a47-41b0-acdc-460b23b24e42" path="/var/lib/kubelet/pods/5b1392b7-3a47-41b0-acdc-460b23b24e42/volumes" Nov 03 22:43:34.412178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:34.412155 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c" path="/var/lib/kubelet/pods/7ecff6f2-9ef9-48a4-9b6a-e1726d7f267c/volumes" Nov 03 22:43:35.352913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.352888 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:35.440042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.439983 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\" (UniqueName: \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\") pod \"6fbf2e61-1345-4f5e-9710-4bddac89800a\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " Nov 03 22:43:35.440251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.440142 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\") pod \"6fbf2e61-1345-4f5e-9710-4bddac89800a\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " Nov 03 22:43:35.440343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.440300 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\") pod \"6fbf2e61-1345-4f5e-9710-4bddac89800a\" (UID: \"6fbf2e61-1345-4f5e-9710-4bddac89800a\") " Nov 03 22:43:35.441454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.441427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca" pod "6fbf2e61-1345-4f5e-9710-4bddac89800a" (UID: "6fbf2e61-1345-4f5e-9710-4bddac89800a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:35.444027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.443473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca" pod "6fbf2e61-1345-4f5e-9710-4bddac89800a" (UID: "6fbf2e61-1345-4f5e-9710-4bddac89800a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:35.444027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.443662 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv" pod "6fbf2e61-1345-4f5e-9710-4bddac89800a" (UID: "6fbf2e61-1345-4f5e-9710-4bddac89800a"). InnerVolumeSpecName "kube-api-access-8nrkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:35.543165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.543130 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:35.543165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.543159 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf2e61-1345-4f5e-9710-4bddac89800a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:35.543165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:35.543169 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nrkv\" (UniqueName: \"kubernetes.io/projected/6fbf2e61-1345-4f5e-9710-4bddac89800a-kube-api-access-8nrkv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:36.205930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:36.205901 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" Nov 03 22:43:36.205930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:36.205922 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx" event={"ID":"6fbf2e61-1345-4f5e-9710-4bddac89800a","Type":"ContainerDied","Data":"1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735"} Nov 03 22:43:36.206238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:36.205968 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5b5e7e343770ac2f880dc745b1ec6387e7d3121411dafc7207f512ac7d6735" Nov 03 22:43:37.050511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:37.050468 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq"] Nov 03 22:43:37.101578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:37.101548 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a18197e-zp6bq"] Nov 03 22:43:38.048831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:38.048800 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4"] Nov 03 22:43:38.097827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:38.097794 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95a276-lvfl4"] Nov 03 22:43:38.409052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:38.409024 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4b8994-0ec6-450a-8f26-250a876eb277" path="/var/lib/kubelet/pods/1b4b8994-0ec6-450a-8f26-250a876eb277/volumes" Nov 03 22:43:38.409508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:38.409494 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b02f11-c59b-4d1d-bb36-c01594488dff" path="/var/lib/kubelet/pods/89b02f11-c59b-4d1d-bb36-c01594488dff/volumes" Nov 03 22:43:40.736065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.736025 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq"] Nov 03 22:43:40.736873 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.736850 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6fbf2e61-1345-4f5e-9710-4bddac89800a" containerName="revoke-keys" Nov 03 22:43:40.737041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.736874 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbf2e61-1345-4f5e-9710-4bddac89800a" containerName="revoke-keys" Nov 03 22:43:40.737119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.737054 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbf2e61-1345-4f5e-9710-4bddac89800a" containerName="revoke-keys" Nov 03 22:43:40.742279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.742259 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.798111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.798078 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq"] Nov 03 22:43:40.816277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.816240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.816467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.816439 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.816566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.816544 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgx2\" (UniqueName: \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.841099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.841064 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:40.841334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.841239 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:43:40.919370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.919333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.919573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.919465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.919573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.919555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgx2\" (UniqueName: \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.920117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.920091 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.922086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.922063 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:40.980808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:40.980779 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgx2\" (UniqueName: \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\") pod \"maas-api-revoke-sub-642bc8a9-2dtdq\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:41.055629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:41.055567 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:41.448462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:41.448436 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq"] Nov 03 22:43:41.449052 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:41.449025 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b934a3a_264b_40ec_84fe_eb616a21bba1.slice/crio-33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd WatchSource:0}: Error finding container 33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd: Status 404 returned error can't find the container with id 33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd Nov 03 22:43:42.255094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:42.255061 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" event={"ID":"2b934a3a-264b-40ec-84fe-eb616a21bba1","Type":"ContainerStarted","Data":"19e4d2e91ad844ec6d8ea3ffbed73b81e67ec4aa7c993534cd0bfd747d99f2ef"} Nov 03 22:43:42.255517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:42.255118 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" event={"ID":"2b934a3a-264b-40ec-84fe-eb616a21bba1","Type":"ContainerStarted","Data":"33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd"} Nov 03 22:43:43.266086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:43.266051 5051 generic.go:356] "Generic (PLEG): container finished" podID="2b934a3a-264b-40ec-84fe-eb616a21bba1" containerID="19e4d2e91ad844ec6d8ea3ffbed73b81e67ec4aa7c993534cd0bfd747d99f2ef" exitCode=0 Nov 03 22:43:43.266609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:43.266105 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" event={"ID":"2b934a3a-264b-40ec-84fe-eb616a21bba1","Type":"ContainerDied","Data":"19e4d2e91ad844ec6d8ea3ffbed73b81e67ec4aa7c993534cd0bfd747d99f2ef"} Nov 03 22:43:44.439932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.439900 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:44.571120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.571034 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\") pod \"2b934a3a-264b-40ec-84fe-eb616a21bba1\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " Nov 03 22:43:44.571297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.571164 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\") pod \"2b934a3a-264b-40ec-84fe-eb616a21bba1\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " Nov 03 22:43:44.571297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.571258 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\" (UniqueName: \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\") pod \"2b934a3a-264b-40ec-84fe-eb616a21bba1\" (UID: \"2b934a3a-264b-40ec-84fe-eb616a21bba1\") " Nov 03 22:43:44.571479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.571453 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca" pod "2b934a3a-264b-40ec-84fe-eb616a21bba1" (UID: "2b934a3a-264b-40ec-84fe-eb616a21bba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:44.572323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.572294 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:44.573696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.573671 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2" pod "2b934a3a-264b-40ec-84fe-eb616a21bba1" (UID: "2b934a3a-264b-40ec-84fe-eb616a21bba1"). InnerVolumeSpecName "kube-api-access-ncgx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:44.573972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.573949 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca" pod "2b934a3a-264b-40ec-84fe-eb616a21bba1" (UID: "2b934a3a-264b-40ec-84fe-eb616a21bba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:44.673931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.673899 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b934a3a-264b-40ec-84fe-eb616a21bba1-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:44.673931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:44.673925 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgx2\" (UniqueName: \"kubernetes.io/projected/2b934a3a-264b-40ec-84fe-eb616a21bba1-kube-api-access-ncgx2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:45.305193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:45.305096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" event={"ID":"2b934a3a-264b-40ec-84fe-eb616a21bba1","Type":"ContainerDied","Data":"33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd"} Nov 03 22:43:45.305453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:45.305433 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bcedf6d85751f6b079540d999fd4c41123defbca3312426c75434982b472dd" Nov 03 22:43:45.305631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:45.305618 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq" Nov 03 22:43:46.612118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:46.612071 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:46.706538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:46.706492 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:46.982074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:46.977355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:46.982074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:46.977657 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:46.996049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:46.993087 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.008039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.006086 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:43:47.107138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.107100 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.108124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.108094 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.164201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.163872 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:43:47.169419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.168696 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.524110 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.524070 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.534341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.534302 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.577194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.577155 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:43:47.577507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.577452 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.597964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.597882 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.614031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.613207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.614031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.613606 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:43:47.635092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.634317 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.709341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.709306 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:47.710818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.710797 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:43:47.871848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:47.853151 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:43:48.040010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.023028 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:48.042412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.037262 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 22:43:48.042911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.042891 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:43:48.043121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.043108 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:43:48.158062 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.157961 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:43:48.158611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.158589 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:43:48.180531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.180503 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:43:48.198212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.198189 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:43:48.339554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.337733 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:43:48.367163 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.367139 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:43:48.384186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.384160 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:43:48.400012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.387725 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:43:48.422208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:48.422132 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:43:49.036759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.036727 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh"] Nov 03 22:43:49.037644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.037621 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2b934a3a-264b-40ec-84fe-eb616a21bba1" containerName="revoke-keys" Nov 03 22:43:49.037644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.037643 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b934a3a-264b-40ec-84fe-eb616a21bba1" containerName="revoke-keys" Nov 03 22:43:49.037807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.037793 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b934a3a-264b-40ec-84fe-eb616a21bba1" containerName="revoke-keys" Nov 03 22:43:49.041670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.041651 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.056852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.056830 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.056976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.056959 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hvw\" (UniqueName: \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.057064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.057006 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.100400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.100363 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:43:49.100583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.100446 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:43:49.101558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.101532 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh"] Nov 03 22:43:49.158052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.157981 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hvw\" (UniqueName: \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.158260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.158077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.158328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.158311 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.159307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.159279 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.160921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.160889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.218706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.218678 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hvw\" (UniqueName: \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\") pod \"maas-api-revoke-sub-13e4fc87-d2cqh\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.354790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.354761 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:49.753765 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:49.753735 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab74d04d_bafc_49e4_a9c0_cfaa24be4626.slice/crio-cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2 WatchSource:0}: Error finding container cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2: Status 404 returned error can't find the container with id cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2 Nov 03 22:43:49.754012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:49.753959 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh"] Nov 03 22:43:50.051851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.051771 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh"] Nov 03 22:43:50.056372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.056346 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72a4bc1-gxswh"] Nov 03 22:43:50.413216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.411535 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff6e5bc-d0ad-4e71-a75b-0946ccef055c" path="/var/lib/kubelet/pods/bff6e5bc-d0ad-4e71-a75b-0946ccef055c/volumes" Nov 03 22:43:50.433177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.433146 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" event={"ID":"ab74d04d-bafc-49e4-a9c0-cfaa24be4626","Type":"ContainerStarted","Data":"3e19f69d94f7cbe95b20c6ee90b3755d557775e7fed6262339611cec8849addd"} Nov 03 22:43:50.433350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.433195 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" event={"ID":"ab74d04d-bafc-49e4-a9c0-cfaa24be4626","Type":"ContainerStarted","Data":"cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2"} Nov 03 22:43:50.550928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:50.550879 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" podStartSLOduration=2.550865843 podStartE2EDuration="2.550865843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:50.547513862 +0000 UTC m=+1304.284858088" watchObservedRunningTime="2026-11-03 22:43:50.550865843 +0000 UTC m=+1304.288210096" Nov 03 22:43:51.443085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:51.443049 5051 generic.go:356] "Generic (PLEG): container finished" podID="ab74d04d-bafc-49e4-a9c0-cfaa24be4626" containerID="3e19f69d94f7cbe95b20c6ee90b3755d557775e7fed6262339611cec8849addd" exitCode=0 Nov 03 22:43:51.443483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:51.443087 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" event={"ID":"ab74d04d-bafc-49e4-a9c0-cfaa24be4626","Type":"ContainerDied","Data":"3e19f69d94f7cbe95b20c6ee90b3755d557775e7fed6262339611cec8849addd"} Nov 03 22:43:51.627354 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:43:51.627320 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:43:51.961515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:51.961478 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d"] Nov 03 22:43:51.965864 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:51.965840 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.005555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.005529 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.005682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.005583 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.005802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.005781 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jr47\" (UniqueName: \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.026335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.026305 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d"] Nov 03 22:43:52.050895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.050870 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66"] Nov 03 22:43:52.076637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.076608 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c57c21-bjx66"] Nov 03 22:43:52.107409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.107380 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jr47\" (UniqueName: \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.107573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.107524 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.107635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.107601 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.108427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.108394 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.110060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.110038 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.168263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.168235 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jr47\" (UniqueName: \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\") pod \"maas-api-revoke-sub-6bd3e364-csr7d\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.283493 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.283422 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:52.423765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.423726 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5946c2b6-c802-41b8-b0ea-2f0edc59a94a" path="/var/lib/kubelet/pods/5946c2b6-c802-41b8-b0ea-2f0edc59a94a/volumes" Nov 03 22:43:52.626808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.626787 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:52.673295 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:52.673265 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4166410_27db_4474_b701_9a5a4492d5db.slice/crio-14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60 WatchSource:0}: Error finding container 14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60: Status 404 returned error can't find the container with id 14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60 Nov 03 22:43:52.674542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.674514 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d"] Nov 03 22:43:52.714950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.714929 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\") pod \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " Nov 03 22:43:52.715166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.715152 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\" (UniqueName: \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\") pod \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " Nov 03 22:43:52.715234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.715202 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\") pod \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\" (UID: \"ab74d04d-bafc-49e4-a9c0-cfaa24be4626\") " Nov 03 22:43:52.715295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.715234 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca" pod "ab74d04d-bafc-49e4-a9c0-cfaa24be4626" (UID: "ab74d04d-bafc-49e4-a9c0-cfaa24be4626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:52.715604 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.715586 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:52.717328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.717300 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw" pod "ab74d04d-bafc-49e4-a9c0-cfaa24be4626" (UID: "ab74d04d-bafc-49e4-a9c0-cfaa24be4626"). InnerVolumeSpecName "kube-api-access-85hvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:52.717408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.717340 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca" pod "ab74d04d-bafc-49e4-a9c0-cfaa24be4626" (UID: "ab74d04d-bafc-49e4-a9c0-cfaa24be4626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:52.816674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.816639 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hvw\" (UniqueName: \"kubernetes.io/projected/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-kube-api-access-85hvw\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:52.816674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:52.816664 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab74d04d-bafc-49e4-a9c0-cfaa24be4626-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:53.463600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.463560 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" event={"ID":"b4166410-27db-4474-b701-9a5a4492d5db","Type":"ContainerStarted","Data":"9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d"} Nov 03 22:43:53.463800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.463626 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" event={"ID":"b4166410-27db-4474-b701-9a5a4492d5db","Type":"ContainerStarted","Data":"14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60"} Nov 03 22:43:53.465262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.465236 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" Nov 03 22:43:53.465262 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.465252 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh" event={"ID":"ab74d04d-bafc-49e4-a9c0-cfaa24be4626","Type":"ContainerDied","Data":"cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2"} Nov 03 22:43:53.465428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.465298 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7337da46f51eafa92a685c03567e7872666115b689f99cca70dcf80e848cd2" Nov 03 22:43:53.581637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:53.581590 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" podStartSLOduration=2.581577502 podStartE2EDuration="2.581577502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:53.57990937 +0000 UTC m=+1307.317253596" watchObservedRunningTime="2026-11-03 22:43:53.581577502 +0000 UTC m=+1307.318921769" Nov 03 22:43:53.767234 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:43:53.767206 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4166410_27db_4474_b701_9a5a4492d5db.slice/crio-conmon-9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4166410_27db_4474_b701_9a5a4492d5db.slice/crio-9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:43:54.476361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:54.476325 5051 generic.go:356] "Generic (PLEG): container finished" podID="b4166410-27db-4474-b701-9a5a4492d5db" containerID="9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d" exitCode=0 Nov 03 22:43:54.476521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:54.476404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" event={"ID":"b4166410-27db-4474-b701-9a5a4492d5db","Type":"ContainerDied","Data":"9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d"} Nov 03 22:43:55.085130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.058469 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs"] Nov 03 22:43:55.085663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.085636 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9590ba86-c28qs"] Nov 03 22:43:55.649942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.649914 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:55.761922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.761846 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\") pod \"b4166410-27db-4474-b701-9a5a4492d5db\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " Nov 03 22:43:55.762095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.761965 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\" (UniqueName: \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\") pod \"b4166410-27db-4474-b701-9a5a4492d5db\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " Nov 03 22:43:55.762095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.762077 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\") pod \"b4166410-27db-4474-b701-9a5a4492d5db\" (UID: \"b4166410-27db-4474-b701-9a5a4492d5db\") " Nov 03 22:43:55.762428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.762405 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca" pod "b4166410-27db-4474-b701-9a5a4492d5db" (UID: "b4166410-27db-4474-b701-9a5a4492d5db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:55.762562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.762544 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:55.764121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.764101 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca" pod "b4166410-27db-4474-b701-9a5a4492d5db" (UID: "b4166410-27db-4474-b701-9a5a4492d5db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:55.764182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.764154 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47" pod "b4166410-27db-4474-b701-9a5a4492d5db" (UID: "b4166410-27db-4474-b701-9a5a4492d5db"). InnerVolumeSpecName "kube-api-access-8jr47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:55.864389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.864355 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jr47\" (UniqueName: \"kubernetes.io/projected/b4166410-27db-4474-b701-9a5a4492d5db-kube-api-access-8jr47\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:55.864389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:55.864385 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4166410-27db-4474-b701-9a5a4492d5db-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:56.171348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.171311 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5"] Nov 03 22:43:56.172311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172292 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b4166410-27db-4474-b701-9a5a4492d5db" containerName="revoke-keys" Nov 03 22:43:56.172358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172315 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4166410-27db-4474-b701-9a5a4492d5db" containerName="revoke-keys" Nov 03 22:43:56.172390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172367 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ab74d04d-bafc-49e4-a9c0-cfaa24be4626" containerName="revoke-keys" Nov 03 22:43:56.172390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172374 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab74d04d-bafc-49e4-a9c0-cfaa24be4626" containerName="revoke-keys" Nov 03 22:43:56.172525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172514 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4166410-27db-4474-b701-9a5a4492d5db" containerName="revoke-keys" Nov 03 22:43:56.172558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.172536 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab74d04d-bafc-49e4-a9c0-cfaa24be4626" containerName="revoke-keys" Nov 03 22:43:56.176962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.176945 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.231296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.231272 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5"] Nov 03 22:43:56.270075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.270047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsxz\" (UniqueName: \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.270341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.270321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.270445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.270424 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.371552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.371527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.371707 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.371620 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsxz\" (UniqueName: \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.371780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.371756 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.372419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.372399 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.374183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.374162 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.409711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.409689 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38231b2e-9b93-4e6e-9cc3-5130fec353e2" path="/var/lib/kubelet/pods/38231b2e-9b93-4e6e-9cc3-5130fec353e2/volumes" Nov 03 22:43:56.432250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.432203 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsxz\" (UniqueName: \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\") pod \"maas-api-revoke-sub-ff2263eb-phzh5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.491463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.491439 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:56.494059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.494028 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" event={"ID":"b4166410-27db-4474-b701-9a5a4492d5db","Type":"ContainerDied","Data":"14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60"} Nov 03 22:43:56.494179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.494090 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14921239de0e05944d90a298dbd8d11a159db9c5f525308021f6c440f2fc7f60" Nov 03 22:43:56.494179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.494041 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d" Nov 03 22:43:56.730587 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.730516 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv"] Nov 03 22:43:56.735750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.735733 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.777487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.777463 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pph8p\" (UniqueName: \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.777592 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.777528 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.777726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.777705 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.793018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.792957 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv"] Nov 03 22:43:56.879111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.879086 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.879239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.879182 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pph8p\" (UniqueName: \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.879299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.879278 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.879784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.879763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.881415 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.881394 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5"] Nov 03 22:43:56.881620 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:56.881604 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e9f5d7_c79e_43ef_83c6_4c6836079db5.slice/crio-058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a WatchSource:0}: Error finding container 058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a: Status 404 returned error can't find the container with id 058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a Nov 03 22:43:56.882275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.882254 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.920357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.920335 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df"] Nov 03 22:43:56.925667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.925653 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:56.939669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.939652 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pph8p\" (UniqueName: \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\") pod \"maas-api-revoke-sub-8f253afa-h9nkv\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:56.981320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.981242 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:56.983102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.981483 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:56.993450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:56.993422 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xkz\" (UniqueName: \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.000309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.000280 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df"] Nov 03 22:43:57.050758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.050729 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:57.094733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.094708 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.094885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.094744 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xkz\" (UniqueName: \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.094885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.094863 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.095684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.095656 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.097511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.097489 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.157592 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.157561 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xkz\" (UniqueName: \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\") pod \"maas-api-revoke-sub-f8ce2642-k27df\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.240159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.240081 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:43:57.440668 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:57.440635 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55446179_8462_49a6_a492_8271e8bb0501.slice/crio-5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5 WatchSource:0}: Error finding container 5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5: Status 404 returned error can't find the container with id 5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5 Nov 03 22:43:57.440796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.440744 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv"] Nov 03 22:43:57.503340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.503282 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" event={"ID":"55446179-8462-49a6-a492-8271e8bb0501","Type":"ContainerStarted","Data":"5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5"} Nov 03 22:43:57.504952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.504928 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" event={"ID":"70e9f5d7-c79e-43ef-83c6-4c6836079db5","Type":"ContainerStarted","Data":"a031b56056df5c01deeb016f497aa1476ead607a245eee4d6bf8bf7d64b2f133"} Nov 03 22:43:57.505147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.504971 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" event={"ID":"70e9f5d7-c79e-43ef-83c6-4c6836079db5","Type":"ContainerStarted","Data":"058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a"} Nov 03 22:43:57.621941 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:43:57.621912 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd868c385_088e_4a61_a6a4_2e1cdc7a0d6c.slice/crio-1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527 WatchSource:0}: Error finding container 1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527: Status 404 returned error can't find the container with id 1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527 Nov 03 22:43:57.622078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.621947 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df"] Nov 03 22:43:57.626205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:57.626168 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" podStartSLOduration=1.62615636 podStartE2EDuration="1.62615636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:57.618948414 +0000 UTC m=+1311.356292637" watchObservedRunningTime="2026-11-03 22:43:57.62615636 +0000 UTC m=+1311.363500586" Nov 03 22:43:58.519258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.519224 5051 generic.go:356] "Generic (PLEG): container finished" podID="70e9f5d7-c79e-43ef-83c6-4c6836079db5" containerID="a031b56056df5c01deeb016f497aa1476ead607a245eee4d6bf8bf7d64b2f133" exitCode=0 Nov 03 22:43:58.519658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.519313 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" event={"ID":"70e9f5d7-c79e-43ef-83c6-4c6836079db5","Type":"ContainerDied","Data":"a031b56056df5c01deeb016f497aa1476ead607a245eee4d6bf8bf7d64b2f133"} Nov 03 22:43:58.521121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.521096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" event={"ID":"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c","Type":"ContainerStarted","Data":"25ded95454b8bb69a168ba5b8202f2fdbb092b7f827cc130f859a17c0571b9b7"} Nov 03 22:43:58.521269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.521141 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" event={"ID":"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c","Type":"ContainerStarted","Data":"1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527"} Nov 03 22:43:58.522649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.522626 5051 generic.go:356] "Generic (PLEG): container finished" podID="55446179-8462-49a6-a492-8271e8bb0501" containerID="6080f1b16f3aa68f92dea59e6b1815f73cc9ac64f767f2719651fbc982b11b17" exitCode=0 Nov 03 22:43:58.522779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.522687 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" event={"ID":"55446179-8462-49a6-a492-8271e8bb0501","Type":"ContainerDied","Data":"6080f1b16f3aa68f92dea59e6b1815f73cc9ac64f767f2719651fbc982b11b17"} Nov 03 22:43:58.868277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:58.868233 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" podStartSLOduration=2.868219623 podStartE2EDuration="2.868219623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:58.864847638 +0000 UTC m=+1312.602191863" watchObservedRunningTime="2026-11-03 22:43:58.868219623 +0000 UTC m=+1312.605564207" Nov 03 22:43:59.531899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.531864 5051 generic.go:356] "Generic (PLEG): container finished" podID="d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" containerID="25ded95454b8bb69a168ba5b8202f2fdbb092b7f827cc130f859a17c0571b9b7" exitCode=0 Nov 03 22:43:59.532388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.531906 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" event={"ID":"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c","Type":"ContainerDied","Data":"25ded95454b8bb69a168ba5b8202f2fdbb092b7f827cc130f859a17c0571b9b7"} Nov 03 22:43:59.738672 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.738643 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:43:59.782769 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.782712 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:43:59.832205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.832167 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\" (UniqueName: \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\") pod \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " Nov 03 22:43:59.832403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.832327 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\") pod \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " Nov 03 22:43:59.832579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.832555 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\") pod \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\" (UID: \"70e9f5d7-c79e-43ef-83c6-4c6836079db5\") " Nov 03 22:43:59.833199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.833171 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca" pod "70e9f5d7-c79e-43ef-83c6-4c6836079db5" (UID: "70e9f5d7-c79e-43ef-83c6-4c6836079db5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:59.833384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.833363 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:59.834543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.834520 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca" pod "70e9f5d7-c79e-43ef-83c6-4c6836079db5" (UID: "70e9f5d7-c79e-43ef-83c6-4c6836079db5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:59.834909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.834894 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz" pod "70e9f5d7-c79e-43ef-83c6-4c6836079db5" (UID: "70e9f5d7-c79e-43ef-83c6-4c6836079db5"). InnerVolumeSpecName "kube-api-access-8nsxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:59.935899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.935773 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\" (UniqueName: \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\") pod \"55446179-8462-49a6-a492-8271e8bb0501\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " Nov 03 22:43:59.936129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.935925 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\") pod \"55446179-8462-49a6-a492-8271e8bb0501\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " Nov 03 22:43:59.936129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.936041 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\") pod \"55446179-8462-49a6-a492-8271e8bb0501\" (UID: \"55446179-8462-49a6-a492-8271e8bb0501\") " Nov 03 22:43:59.936693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.936630 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsxz\" (UniqueName: \"kubernetes.io/projected/70e9f5d7-c79e-43ef-83c6-4c6836079db5-kube-api-access-8nsxz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:59.936693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.936660 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e9f5d7-c79e-43ef-83c6-4c6836079db5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:43:59.937447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.937420 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca" pod "55446179-8462-49a6-a492-8271e8bb0501" (UID: "55446179-8462-49a6-a492-8271e8bb0501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:59.939498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.939473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca" pod "55446179-8462-49a6-a492-8271e8bb0501" (UID: "55446179-8462-49a6-a492-8271e8bb0501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:59.945688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:43:59.945663 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p" pod "55446179-8462-49a6-a492-8271e8bb0501" (UID: "55446179-8462-49a6-a492-8271e8bb0501"). InnerVolumeSpecName "kube-api-access-pph8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:00.038941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.038857 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pph8p\" (UniqueName: \"kubernetes.io/projected/55446179-8462-49a6-a492-8271e8bb0501-kube-api-access-pph8p\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:00.038941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.038892 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55446179-8462-49a6-a492-8271e8bb0501-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:00.038941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.038907 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55446179-8462-49a6-a492-8271e8bb0501-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:00.543053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.543019 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" event={"ID":"70e9f5d7-c79e-43ef-83c6-4c6836079db5","Type":"ContainerDied","Data":"058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a"} Nov 03 22:44:00.543457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.543064 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5" Nov 03 22:44:00.543457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.543070 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058ccbb2299f38b0636e4ad23c2b430ece22f62fb6e365b7d28e14f51ddb973a" Nov 03 22:44:00.544662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.544640 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" Nov 03 22:44:00.544784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.544665 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv" event={"ID":"55446179-8462-49a6-a492-8271e8bb0501","Type":"ContainerDied","Data":"5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5"} Nov 03 22:44:00.544784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.544698 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4b0977d1613b67a226e3cf5d33da897661e2b4cc66769d63c54a423f13a7c5" Nov 03 22:44:00.727969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.727930 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:44:00.851926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.851895 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\") pod \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " Nov 03 22:44:00.852185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.852163 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\") pod \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " Nov 03 22:44:00.852541 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.852491 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca" pod "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" (UID: "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:00.852637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.852579 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\" (UniqueName: \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\") pod \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\" (UID: \"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c\") " Nov 03 22:44:00.853253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.853228 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:00.855119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.855096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca" pod "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" (UID: "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:00.860153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.860128 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz" pod "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" (UID: "d868c385-088e-4a61-a6a4-2e1cdc7a0d6c"). InnerVolumeSpecName "kube-api-access-p6xkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:00.954466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.954382 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:00.954466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:00.954410 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xkz\" (UniqueName: \"kubernetes.io/projected/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c-kube-api-access-p6xkz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:01.553759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:01.553728 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" Nov 03 22:44:01.553759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:01.553751 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df" event={"ID":"d868c385-088e-4a61-a6a4-2e1cdc7a0d6c","Type":"ContainerDied","Data":"1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527"} Nov 03 22:44:01.554212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:01.553797 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1655ff99381f28bbde49e2dc3c9411527cbc7d6288c06aecc5d3e9609eca1527" Nov 03 22:44:03.052039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.051971 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc"] Nov 03 22:44:03.102287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.102255 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx"] Nov 03 22:44:03.106828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.106802 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603aa8b5-q42mc"] Nov 03 22:44:03.111167 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.111141 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f7aa5b-gs4hx"] Nov 03 22:44:03.899725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.899695 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:44:03.900382 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900366 5051 cpu_manager.go:414] "containerMap: removing container" podUID="70e9f5d7-c79e-43ef-83c6-4c6836079db5" containerName="revoke-keys" Nov 03 22:44:03.900453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900383 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e9f5d7-c79e-43ef-83c6-4c6836079db5" containerName="revoke-keys" Nov 03 22:44:03.900453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900397 5051 cpu_manager.go:414] "containerMap: removing container" podUID="55446179-8462-49a6-a492-8271e8bb0501" containerName="revoke-keys" Nov 03 22:44:03.900453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900402 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55446179-8462-49a6-a492-8271e8bb0501" containerName="revoke-keys" Nov 03 22:44:03.900544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900458 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" containerName="revoke-keys" Nov 03 22:44:03.900544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900464 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" containerName="revoke-keys" Nov 03 22:44:03.900602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900569 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" containerName="revoke-keys" Nov 03 22:44:03.900602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900581 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e9f5d7-c79e-43ef-83c6-4c6836079db5" containerName="revoke-keys" Nov 03 22:44:03.900602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.900592 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="55446179-8462-49a6-a492-8271e8bb0501" containerName="revoke-keys" Nov 03 22:44:03.904451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.904433 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.949501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.949479 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:44:03.953760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.953743 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:03.995488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995456 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.995631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995523 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.995631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995612 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.995733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995651 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.995733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995705 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.995955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995936 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.996043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.995981 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.996091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.996074 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x77\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:03.996176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:03.996158 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.006895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.005103 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:44:04.055742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.055706 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:44:04.097434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097399 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.097632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097464 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.097632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097497 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.097632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097524 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.097632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097554 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x77\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.097887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097588 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.097887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097787 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.097887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097849 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.097887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097861 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.098144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097913 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.097950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.098144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098087 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.098144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.098384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098219 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098249 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7qm\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098278 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.098384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098324 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098426 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.098640 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.098581 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.099034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.099012 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.100093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.100071 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.100549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.100528 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.103307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.103280 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.103735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.103708 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.120036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.120006 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:44:04.131346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.131316 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.164407 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.164327 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.165263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.165233 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x77\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.197009 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.192339 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.213954 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214021 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7qm\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214069 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214155 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214259 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214383 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214452 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.214482 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.217020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.215384 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.219618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.219368 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.220666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.220639 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.220840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.220822 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:04.221446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.221420 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.221704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.221682 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.240835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.239320 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.247384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.247358 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.285426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.285262 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.285426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.285378 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7qm\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.316619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316579 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.316792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316698 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.316792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316754 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.316909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316844 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.316909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316888 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.317061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.316952 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.317061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.317025 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.317177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.317070 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.317177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.317154 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vxq\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.411870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.411842 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11dedf8-dbca-4fca-949f-f9eae9007cee" path="/var/lib/kubelet/pods/a11dedf8-dbca-4fca-949f-f9eae9007cee/volumes" Nov 03 22:44:04.412296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.412276 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60dfae6-adb8-4e77-8c18-6aa9a4be00a3" path="/var/lib/kubelet/pods/a60dfae6-adb8-4e77-8c18-6aa9a4be00a3/volumes" Nov 03 22:44:04.418573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418505 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vxq\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.418573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418557 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.418757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418640 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.418757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418681 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.418880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418858 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.418948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418915 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.418969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419040 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419054 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419121 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419165 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419418 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419489 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.419530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.419510 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.421326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.421308 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.421468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.421444 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.479006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.478952 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.479160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.479026 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vxq\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.567729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.567698 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:04.676795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.676762 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:44:04.677132 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:04.677106 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414bff10_4e37_4ef8_be4c_11859f72ae10.slice/crio-0936044e150cca46b0f63cce8ac94c279918d871a5ac404a8355fd555d319608 WatchSource:0}: Error finding container 0936044e150cca46b0f63cce8ac94c279918d871a5ac404a8355fd555d319608: Status 404 returned error can't find the container with id 0936044e150cca46b0f63cce8ac94c279918d871a5ac404a8355fd555d319608 Nov 03 22:44:04.680147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.680109 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.680285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.680183 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.680285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.680219 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.755684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.755660 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:04.993155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.993129 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:44:04.994721 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:04.994692 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3c4c30_bf0f_4db7_8024_d7cbf0e90f28.slice/crio-1e17ece8aafe066d39b2a746f9c1a7da9a39c1335fc97a76e445a92f60a80c6e WatchSource:0}: Error finding container 1e17ece8aafe066d39b2a746f9c1a7da9a39c1335fc97a76e445a92f60a80c6e: Status 404 returned error can't find the container with id 1e17ece8aafe066d39b2a746f9c1a7da9a39c1335fc97a76e445a92f60a80c6e Nov 03 22:44:04.997662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.997626 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.997765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.997695 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.997765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:04.997727 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:05.185034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.185003 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:44:05.185516 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:05.185487 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf393dcee_c79b_4d3d_9cf5_906209391d94.slice/crio-a52e690c1c9b70cb46da1693a132eef7cc9821dd45690a5cba522d4fd74cb83b WatchSource:0}: Error finding container a52e690c1c9b70cb46da1693a132eef7cc9821dd45690a5cba522d4fd74cb83b: Status 404 returned error can't find the container with id a52e690c1c9b70cb46da1693a132eef7cc9821dd45690a5cba522d4fd74cb83b Nov 03 22:44:05.188193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.188158 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:05.188318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.188244 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:05.188318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.188289 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:05.613225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.603947 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" event={"ID":"414bff10-4e37-4ef8-be4c-11859f72ae10","Type":"ContainerStarted","Data":"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946"} Nov 03 22:44:05.613225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.604034 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" event={"ID":"414bff10-4e37-4ef8-be4c-11859f72ae10","Type":"ContainerStarted","Data":"0936044e150cca46b0f63cce8ac94c279918d871a5ac404a8355fd555d319608"} Nov 03 22:44:05.631338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.631294 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" event={"ID":"f393dcee-c79b-4d3d-9cf5-906209391d94","Type":"ContainerStarted","Data":"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24"} Nov 03 22:44:05.631526 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.631362 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" event={"ID":"f393dcee-c79b-4d3d-9cf5-906209391d94","Type":"ContainerStarted","Data":"a52e690c1c9b70cb46da1693a132eef7cc9821dd45690a5cba522d4fd74cb83b"} Nov 03 22:44:05.633462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.633428 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" event={"ID":"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28","Type":"ContainerStarted","Data":"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a"} Nov 03 22:44:05.633593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.633484 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" event={"ID":"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28","Type":"ContainerStarted","Data":"1e17ece8aafe066d39b2a746f9c1a7da9a39c1335fc97a76e445a92f60a80c6e"} Nov 03 22:44:05.755199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.755146 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podStartSLOduration=2.755132181 podStartE2EDuration="2.755132181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:05.753085575 +0000 UTC m=+1319.490429798" watchObservedRunningTime="2026-11-03 22:44:05.755132181 +0000 UTC m=+1319.492476404" Nov 03 22:44:05.760113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:05.760083 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:06.028126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.027982 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podStartSLOduration=2.027961453 podStartE2EDuration="2.027961453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:06.025299607 +0000 UTC m=+1319.762643831" watchObservedRunningTime="2026-11-03 22:44:06.027961453 +0000 UTC m=+1319.765305678" Nov 03 22:44:06.039492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.038299 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podStartSLOduration=3.038275814 podStartE2EDuration="3.038275814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:05.898419009 +0000 UTC m=+1319.635763247" watchObservedRunningTime="2026-11-03 22:44:06.038275814 +0000 UTC m=+1319.775620059" Nov 03 22:44:06.231409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.231372 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:06.237060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.232131 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:06.237270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.237110 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:06.573030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.569368 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:06.596016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.577119 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:06.596016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.577181 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:06.764017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.761715 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:06.764017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:06.761777 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:07.115656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.111883 5051 scope.go:122] "RemoveContainer" containerID="d5e5e218eaf2c8a85eb561f7f9a4e465f4aecb68245ef20fb70f5a9756f8dc7b" Nov 03 22:44:07.217742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.216832 5051 scope.go:122] "RemoveContainer" containerID="5f26f959bfe367b491ca48f367b3981b36c81bb82ecba124d92e48e56a428330" Nov 03 22:44:07.245019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.241520 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:07.245019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.241582 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:07.337015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.331183 5051 scope.go:122] "RemoveContainer" containerID="f9600a353f35e2a4b423717990c616bc7de79c92a382b6f28a77d689d2d22356" Nov 03 22:44:07.378092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.378063 5051 scope.go:122] "RemoveContainer" containerID="8abfd020c53924715eb6a5d1d63cf7d0872c1f4e6f5f864683c6cc74bfe8a8d9" Nov 03 22:44:07.583023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.582396 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:07.583023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.582461 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:07.787075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.786137 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:07.787075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:07.786220 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:08.228486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.228428 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:08.228742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.228715 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:08.623016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.607135 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:08.623016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.607192 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:08.758017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.757478 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:08.758017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:08.757543 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:09.233016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.229486 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:09.233016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.229572 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:09.580117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.579684 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:09.580117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.579744 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:09.777666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.777626 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:09.778331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:09.778300 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:10.225016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.224525 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:10.225016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.224609 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:10.571723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.571603 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:10.572487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.571947 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:10.639403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.634921 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5"] Nov 03 22:44:10.655534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.655495 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.698067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.698033 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5"] Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.838338 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.838475 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.839603 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.839658 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.840921 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.841034 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wxv\" (UniqueName: \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.841131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.841084 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.943150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.943122 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.943403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.943386 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wxv\" (UniqueName: \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.943549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.943534 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.944079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.944058 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:10.945679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:10.945660 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:11.023778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.023705 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wxv\" (UniqueName: \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\") pod \"maas-api-revoke-sub-0282acde-wqnw5\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:11.029247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.029220 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:11.231017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.221712 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:44:11.231017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.221770 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:44:11.508213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.508179 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5"] Nov 03 22:44:11.509797 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:11.509774 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34642db4_9235_4535_b360_c81d2168b284.slice/crio-8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703 WatchSource:0}: Error finding container 8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703: Status 404 returned error can't find the container with id 8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703 Nov 03 22:44:11.568682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.568654 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:44:11.568798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.568709 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:44:11.678297 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.678267 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7"] Nov 03 22:44:11.684637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.684610 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.738056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.738018 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7"] Nov 03 22:44:11.765263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.765192 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:44:11.765504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.765465 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:44:11.866650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.866626 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splps\" (UniqueName: \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.867351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.867328 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.867533 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.867519 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.953006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.952963 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" event={"ID":"34642db4-9235-4535-b360-c81d2168b284","Type":"ContainerStarted","Data":"50ccee7571bdada1f0fa45503aae1d2bcc0ffd7dad67c2833b45f8659d42f2d8"} Nov 03 22:44:11.953158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.953047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" event={"ID":"34642db4-9235-4535-b360-c81d2168b284","Type":"ContainerStarted","Data":"8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703"} Nov 03 22:44:11.972016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.968979 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splps\" (UniqueName: \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.972016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.969145 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.972016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.969234 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.972016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.970247 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:11.973685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:11.973660 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:12.032133 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.032050 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-splps\" (UniqueName: \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\") pod \"maas-api-revoke-sub-10a9d704-t8nw7\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:12.070762 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.070027 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" podStartSLOduration=2.069969793 podStartE2EDuration="2.069969793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:12.069191364 +0000 UTC m=+1325.806535587" watchObservedRunningTime="2026-11-03 22:44:12.069969793 +0000 UTC m=+1325.807314017" Nov 03 22:44:12.304282 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.304189 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:12.533921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.533501 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:12.535292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.535263 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:12.546240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.546215 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:44:12.561481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.561404 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj"] Nov 03 22:44:12.573581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.573562 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.667339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.667298 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj"] Nov 03 22:44:12.700912 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.700152 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.701534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.701513 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7l4\" (UniqueName: \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.701907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.701879 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.748033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.747775 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:12.748802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.748775 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:12.750706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.750690 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:44:12.770098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.770068 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7"] Nov 03 22:44:12.805020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.803458 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7l4\" (UniqueName: \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.805020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.803583 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.805020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.803626 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.805020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.804476 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.811178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.811153 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.871264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.871194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7l4\" (UniqueName: \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\") pod \"maas-api-revoke-sub-20cc7e34-ljgcj\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:12.969052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.968984 5051 generic.go:356] "Generic (PLEG): container finished" podID="34642db4-9235-4535-b360-c81d2168b284" containerID="50ccee7571bdada1f0fa45503aae1d2bcc0ffd7dad67c2833b45f8659d42f2d8" exitCode=0 Nov 03 22:44:12.969294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.969042 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" event={"ID":"34642db4-9235-4535-b360-c81d2168b284","Type":"ContainerDied","Data":"50ccee7571bdada1f0fa45503aae1d2bcc0ffd7dad67c2833b45f8659d42f2d8"} Nov 03 22:44:12.971418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.971396 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" event={"ID":"8380d0c7-517d-48a2-9ad7-e5951a060780","Type":"ContainerStarted","Data":"d17101e886ad9188aed3607c73edfc77c5ca05c44a5744156f067f3f12ec4f88"} Nov 03 22:44:12.981011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:12.979150 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:13.057642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.056509 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk"] Nov 03 22:44:13.081153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.081068 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-412f84f8-z5xlk"] Nov 03 22:44:13.103134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.103106 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:13.105419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.105396 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:13.115842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.115638 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:44:13.447777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.447749 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85"] Nov 03 22:44:13.458113 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:13.458078 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db3dff_af6c_482e_92fe_54c895cece64.slice/crio-f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54 WatchSource:0}: Error finding container f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54: Status 404 returned error can't find the container with id f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54 Nov 03 22:44:13.461373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.461351 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj"] Nov 03 22:44:13.461546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.461422 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.518218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.518191 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85"] Nov 03 22:44:13.520793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.520766 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh7x\" (UniqueName: \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.520976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.520939 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.521212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.521189 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.622969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.622887 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh7x\" (UniqueName: \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.623127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.623044 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.623171 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.623128 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.623736 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.623718 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.657470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.657423 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.683701 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.683675 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh7x\" (UniqueName: \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\") pod \"maas-api-revoke-sub-61931a53-ssf85\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:13.822456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:13.822428 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:14.009178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.009133 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" event={"ID":"8380d0c7-517d-48a2-9ad7-e5951a060780","Type":"ContainerStarted","Data":"6b1d915d5753083642c3c182cd6d21bc139c8c3f9f1feab1e953f01bf8758eb9"} Nov 03 22:44:14.011250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.011223 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" event={"ID":"b4db3dff-af6c-482e-92fe-54c895cece64","Type":"ContainerStarted","Data":"6519e03646f38fb3fbeee9b810aedbe7e7b3b36335fe2f0d3e0effe25794fbab"} Nov 03 22:44:14.011364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.011275 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" event={"ID":"b4db3dff-af6c-482e-92fe-54c895cece64","Type":"ContainerStarted","Data":"f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54"} Nov 03 22:44:14.135723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.134155 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" podStartSLOduration=3.134135915 podStartE2EDuration="3.134135915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:14.129285142 +0000 UTC m=+1327.866629368" watchObservedRunningTime="2026-11-03 22:44:14.134135915 +0000 UTC m=+1327.871480140" Nov 03 22:44:14.233601 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.233578 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:14.234411 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.234392 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\") pod \"34642db4-9235-4535-b360-c81d2168b284\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " Nov 03 22:44:14.234514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.234501 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\") pod \"34642db4-9235-4535-b360-c81d2168b284\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " Nov 03 22:44:14.234630 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.234620 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\" (UniqueName: \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\") pod \"34642db4-9235-4535-b360-c81d2168b284\" (UID: \"34642db4-9235-4535-b360-c81d2168b284\") " Nov 03 22:44:14.234843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.234818 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca" pod "34642db4-9235-4535-b360-c81d2168b284" (UID: "34642db4-9235-4535-b360-c81d2168b284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:14.235088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.235068 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34642db4-9235-4535-b360-c81d2168b284-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:14.236643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.236622 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca" pod "34642db4-9235-4535-b360-c81d2168b284" (UID: "34642db4-9235-4535-b360-c81d2168b284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:14.237759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.237735 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv" pod "34642db4-9235-4535-b360-c81d2168b284" (UID: "34642db4-9235-4535-b360-c81d2168b284"). InnerVolumeSpecName "kube-api-access-x8wxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:14.245630 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.245590 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" podStartSLOduration=2.245579199 podStartE2EDuration="2.245579199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:14.244379793 +0000 UTC m=+1327.981724017" watchObservedRunningTime="2026-11-03 22:44:14.245579199 +0000 UTC m=+1327.982923422" Nov 03 22:44:14.252201 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:14.252180 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99c0a29_be2a_4799_841d_ea0eaeadbde9.slice/crio-aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86 WatchSource:0}: Error finding container aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86: Status 404 returned error can't find the container with id aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86 Nov 03 22:44:14.255791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.255772 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85"] Nov 03 22:44:14.336160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.336133 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wxv\" (UniqueName: \"kubernetes.io/projected/34642db4-9235-4535-b360-c81d2168b284-kube-api-access-x8wxv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:14.336160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.336158 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34642db4-9235-4535-b360-c81d2168b284-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:14.418319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.418285 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d591b68-3bbc-4432-9bf2-57a089da8872" path="/var/lib/kubelet/pods/2d591b68-3bbc-4432-9bf2-57a089da8872/volumes" Nov 03 22:44:14.723375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.723341 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv"] Nov 03 22:44:14.724140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.724119 5051 cpu_manager.go:414] "containerMap: removing container" podUID="34642db4-9235-4535-b360-c81d2168b284" containerName="revoke-keys" Nov 03 22:44:14.724140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.724140 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34642db4-9235-4535-b360-c81d2168b284" containerName="revoke-keys" Nov 03 22:44:14.724318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.724301 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="34642db4-9235-4535-b360-c81d2168b284" containerName="revoke-keys" Nov 03 22:44:14.728063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.728045 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.740689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.740666 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.740906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.740890 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.740948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.740937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frnc\" (UniqueName: \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.784122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.784062 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv"] Nov 03 22:44:14.842379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.842348 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.842555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.842435 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frnc\" (UniqueName: \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.842678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.842589 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.843525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.843496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.845034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.845016 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:14.910602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:14.910571 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frnc\" (UniqueName: \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\") pod \"maas-api-revoke-sub-38c5768e-m7btv\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:15.026005 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.025955 5051 generic.go:356] "Generic (PLEG): container finished" podID="8380d0c7-517d-48a2-9ad7-e5951a060780" containerID="6b1d915d5753083642c3c182cd6d21bc139c8c3f9f1feab1e953f01bf8758eb9" exitCode=0 Nov 03 22:44:15.026511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.026047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" event={"ID":"8380d0c7-517d-48a2-9ad7-e5951a060780","Type":"ContainerDied","Data":"6b1d915d5753083642c3c182cd6d21bc139c8c3f9f1feab1e953f01bf8758eb9"} Nov 03 22:44:15.027965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.027938 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" event={"ID":"a99c0a29-be2a-4799-841d-ea0eaeadbde9","Type":"ContainerStarted","Data":"3cba564b67dc7bd7787e9b19169384a4803da9f2a66052c8eb3fff575a393f8b"} Nov 03 22:44:15.028157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.028120 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" event={"ID":"a99c0a29-be2a-4799-841d-ea0eaeadbde9","Type":"ContainerStarted","Data":"aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86"} Nov 03 22:44:15.029802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.029784 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" Nov 03 22:44:15.029937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.029789 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5" event={"ID":"34642db4-9235-4535-b360-c81d2168b284","Type":"ContainerDied","Data":"8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703"} Nov 03 22:44:15.030120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.029960 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0c2ef5ea6239e936efd438656044a702e9ab5fe00e12de869ad2715b593703" Nov 03 22:44:15.031667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.031646 5051 generic.go:356] "Generic (PLEG): container finished" podID="b4db3dff-af6c-482e-92fe-54c895cece64" containerID="6519e03646f38fb3fbeee9b810aedbe7e7b3b36335fe2f0d3e0effe25794fbab" exitCode=0 Nov 03 22:44:15.031794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.031677 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" event={"ID":"b4db3dff-af6c-482e-92fe-54c895cece64","Type":"ContainerDied","Data":"6519e03646f38fb3fbeee9b810aedbe7e7b3b36335fe2f0d3e0effe25794fbab"} Nov 03 22:44:15.041219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.041159 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:15.427658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.427597 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" podStartSLOduration=2.4275858120000002 podStartE2EDuration="2.427585812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:15.425284426 +0000 UTC m=+1329.162628666" watchObservedRunningTime="2026-11-03 22:44:15.427585812 +0000 UTC m=+1329.164930037" Nov 03 22:44:15.439258 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:15.439231 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eaaa1c_9772_4fda_a09f_1d8e84024a92.slice/crio-46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005 WatchSource:0}: Error finding container 46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005: Status 404 returned error can't find the container with id 46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005 Nov 03 22:44:15.439382 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:15.439355 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv"] Nov 03 22:44:16.041052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.040929 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" event={"ID":"c7eaaa1c-9772-4fda-a09f-1d8e84024a92","Type":"ContainerStarted","Data":"06c32d764c98da70c14968ae3dad23184887b7f859496a184da3ae497cc09760"} Nov 03 22:44:16.041052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.041023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" event={"ID":"c7eaaa1c-9772-4fda-a09f-1d8e84024a92","Type":"ContainerStarted","Data":"46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005"} Nov 03 22:44:16.043034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.042975 5051 generic.go:356] "Generic (PLEG): container finished" podID="a99c0a29-be2a-4799-841d-ea0eaeadbde9" containerID="3cba564b67dc7bd7787e9b19169384a4803da9f2a66052c8eb3fff575a393f8b" exitCode=0 Nov 03 22:44:16.043156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.043025 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" event={"ID":"a99c0a29-be2a-4799-841d-ea0eaeadbde9","Type":"ContainerDied","Data":"3cba564b67dc7bd7787e9b19169384a4803da9f2a66052c8eb3fff575a393f8b"} Nov 03 22:44:16.158280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.158179 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" podStartSLOduration=2.158159912 podStartE2EDuration="2.158159912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:16.155750818 +0000 UTC m=+1329.893095042" watchObservedRunningTime="2026-11-03 22:44:16.158159912 +0000 UTC m=+1329.895504136" Nov 03 22:44:16.237677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.237648 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:16.265919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.265827 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\" (UniqueName: \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\") pod \"8380d0c7-517d-48a2-9ad7-e5951a060780\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " Nov 03 22:44:16.265919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.265883 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\") pod \"8380d0c7-517d-48a2-9ad7-e5951a060780\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " Nov 03 22:44:16.266253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.266183 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\") pod \"8380d0c7-517d-48a2-9ad7-e5951a060780\" (UID: \"8380d0c7-517d-48a2-9ad7-e5951a060780\") " Nov 03 22:44:16.266815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.266793 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca" pod "8380d0c7-517d-48a2-9ad7-e5951a060780" (UID: "8380d0c7-517d-48a2-9ad7-e5951a060780"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:16.268273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.268251 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca" pod "8380d0c7-517d-48a2-9ad7-e5951a060780" (UID: "8380d0c7-517d-48a2-9ad7-e5951a060780"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:16.268609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.268586 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps" pod "8380d0c7-517d-48a2-9ad7-e5951a060780" (UID: "8380d0c7-517d-48a2-9ad7-e5951a060780"). InnerVolumeSpecName "kube-api-access-splps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:16.290476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.290451 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:16.368078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.368051 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\" (UniqueName: \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\") pod \"b4db3dff-af6c-482e-92fe-54c895cece64\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " Nov 03 22:44:16.368245 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.368229 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\") pod \"b4db3dff-af6c-482e-92fe-54c895cece64\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " Nov 03 22:44:16.368378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.368354 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\") pod \"b4db3dff-af6c-482e-92fe-54c895cece64\" (UID: \"b4db3dff-af6c-482e-92fe-54c895cece64\") " Nov 03 22:44:16.368680 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.368617 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca" pod "b4db3dff-af6c-482e-92fe-54c895cece64" (UID: "b4db3dff-af6c-482e-92fe-54c895cece64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:16.369205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.369184 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:16.369313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.369211 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:16.369313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.369227 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splps\" (UniqueName: \"kubernetes.io/projected/8380d0c7-517d-48a2-9ad7-e5951a060780-kube-api-access-splps\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:16.369313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.369241 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380d0c7-517d-48a2-9ad7-e5951a060780-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:16.370348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.370310 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4" pod "b4db3dff-af6c-482e-92fe-54c895cece64" (UID: "b4db3dff-af6c-482e-92fe-54c895cece64"). InnerVolumeSpecName "kube-api-access-zm7l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:16.371007 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.370970 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca" pod "b4db3dff-af6c-482e-92fe-54c895cece64" (UID: "b4db3dff-af6c-482e-92fe-54c895cece64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:16.471212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.471177 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7l4\" (UniqueName: \"kubernetes.io/projected/b4db3dff-af6c-482e-92fe-54c895cece64-kube-api-access-zm7l4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:16.471357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:16.471294 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db3dff-af6c-482e-92fe-54c895cece64-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:17.054834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.054804 5051 generic.go:356] "Generic (PLEG): container finished" podID="c7eaaa1c-9772-4fda-a09f-1d8e84024a92" containerID="06c32d764c98da70c14968ae3dad23184887b7f859496a184da3ae497cc09760" exitCode=0 Nov 03 22:44:17.055275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.054882 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" event={"ID":"c7eaaa1c-9772-4fda-a09f-1d8e84024a92","Type":"ContainerDied","Data":"06c32d764c98da70c14968ae3dad23184887b7f859496a184da3ae497cc09760"} Nov 03 22:44:17.056417 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.056395 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" Nov 03 22:44:17.056551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.056420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj" event={"ID":"b4db3dff-af6c-482e-92fe-54c895cece64","Type":"ContainerDied","Data":"f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54"} Nov 03 22:44:17.056551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.056460 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3938bb80bb8b5909e603b888522c6081ca47d14679cdc18beeffe69d1105c54" Nov 03 22:44:17.057915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.057892 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" event={"ID":"8380d0c7-517d-48a2-9ad7-e5951a060780","Type":"ContainerDied","Data":"d17101e886ad9188aed3607c73edfc77c5ca05c44a5744156f067f3f12ec4f88"} Nov 03 22:44:17.057915 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.057913 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7" Nov 03 22:44:17.058178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.057937 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17101e886ad9188aed3607c73edfc77c5ca05c44a5744156f067f3f12ec4f88" Nov 03 22:44:17.253380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.253355 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:17.282474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.282444 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\") pod \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " Nov 03 22:44:17.282643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.282522 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\" (UniqueName: \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\") pod \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " Nov 03 22:44:17.282643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.282593 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\") pod \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\" (UID: \"a99c0a29-be2a-4799-841d-ea0eaeadbde9\") " Nov 03 22:44:17.282965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.282922 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca" pod "a99c0a29-be2a-4799-841d-ea0eaeadbde9" (UID: "a99c0a29-be2a-4799-841d-ea0eaeadbde9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:17.283536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.283511 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:17.284703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.284682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca" pod "a99c0a29-be2a-4799-841d-ea0eaeadbde9" (UID: "a99c0a29-be2a-4799-841d-ea0eaeadbde9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:17.284803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.284771 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x" pod "a99c0a29-be2a-4799-841d-ea0eaeadbde9" (UID: "a99c0a29-be2a-4799-841d-ea0eaeadbde9"). InnerVolumeSpecName "kube-api-access-rsh7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:17.385472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.385430 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99c0a29-be2a-4799-841d-ea0eaeadbde9-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:17.385472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:17.385469 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh7x\" (UniqueName: \"kubernetes.io/projected/a99c0a29-be2a-4799-841d-ea0eaeadbde9-kube-api-access-rsh7x\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:18.051051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.051017 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn"] Nov 03 22:44:18.070471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.070443 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" Nov 03 22:44:18.070923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.070471 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85" event={"ID":"a99c0a29-be2a-4799-841d-ea0eaeadbde9","Type":"ContainerDied","Data":"aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86"} Nov 03 22:44:18.070923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.070522 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaa93539c1b031bc13129fea56db79d2ed0d474368934e94d5335ae8cb99dd86" Nov 03 22:44:18.101522 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.101487 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da02af2f-qwwvn"] Nov 03 22:44:18.247926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.247904 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:18.299444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.299407 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\" (UniqueName: \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\") pod \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " Nov 03 22:44:18.299676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.299634 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\") pod \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " Nov 03 22:44:18.299738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.299718 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\") pod \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\" (UID: \"c7eaaa1c-9772-4fda-a09f-1d8e84024a92\") " Nov 03 22:44:18.300143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.300116 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca" pod "c7eaaa1c-9772-4fda-a09f-1d8e84024a92" (UID: "c7eaaa1c-9772-4fda-a09f-1d8e84024a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:18.300458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.300440 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:18.302281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.302222 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc" pod "c7eaaa1c-9772-4fda-a09f-1d8e84024a92" (UID: "c7eaaa1c-9772-4fda-a09f-1d8e84024a92"). InnerVolumeSpecName "kube-api-access-4frnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:18.302393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.302272 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca" pod "c7eaaa1c-9772-4fda-a09f-1d8e84024a92" (UID: "c7eaaa1c-9772-4fda-a09f-1d8e84024a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:18.402078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.402036 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:18.402078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.402068 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frnc\" (UniqueName: \"kubernetes.io/projected/c7eaaa1c-9772-4fda-a09f-1d8e84024a92-kube-api-access-4frnc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:18.416865 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:18.416830 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edd9358-77b3-431c-8f9a-58a32a85e585" path="/var/lib/kubelet/pods/3edd9358-77b3-431c-8f9a-58a32a85e585/volumes" Nov 03 22:44:19.080652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:19.080622 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" Nov 03 22:44:19.080652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:19.080644 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv" event={"ID":"c7eaaa1c-9772-4fda-a09f-1d8e84024a92","Type":"ContainerDied","Data":"46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005"} Nov 03 22:44:19.081129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:19.080713 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46fee2791f02ad888f8338800a7d16f5ad2d589c2a667dc20d5512003f829005" Nov 03 22:44:21.086013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.085097 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw"] Nov 03 22:44:21.191213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.175327 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71852cd-k89tw"] Nov 03 22:44:21.399371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.395112 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.399371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.396305 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.425456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.423096 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.425456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.423466 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.533026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.532292 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:44:21.613015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.612085 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.613015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.612473 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 03 22:44:21.636102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.636066 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.832183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.832082 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.978129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.978084 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.979477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:21.979448 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.995290 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:44:21.995259 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:44:22.069664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.069634 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn"] Nov 03 22:44:22.070930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.070904 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8380d0c7-517d-48a2-9ad7-e5951a060780" containerName="revoke-keys" Nov 03 22:44:22.071196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071180 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8380d0c7-517d-48a2-9ad7-e5951a060780" containerName="revoke-keys" Nov 03 22:44:22.071358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071345 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a99c0a29-be2a-4799-841d-ea0eaeadbde9" containerName="revoke-keys" Nov 03 22:44:22.071447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071436 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99c0a29-be2a-4799-841d-ea0eaeadbde9" containerName="revoke-keys" Nov 03 22:44:22.071555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071544 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b4db3dff-af6c-482e-92fe-54c895cece64" containerName="revoke-keys" Nov 03 22:44:22.071642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071630 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4db3dff-af6c-482e-92fe-54c895cece64" containerName="revoke-keys" Nov 03 22:44:22.071729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071720 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c7eaaa1c-9772-4fda-a09f-1d8e84024a92" containerName="revoke-keys" Nov 03 22:44:22.071805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.071795 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eaaa1c-9772-4fda-a09f-1d8e84024a92" containerName="revoke-keys" Nov 03 22:44:22.113194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.113166 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4db3dff-af6c-482e-92fe-54c895cece64" containerName="revoke-keys" Nov 03 22:44:22.113759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.113742 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eaaa1c-9772-4fda-a09f-1d8e84024a92" containerName="revoke-keys" Nov 03 22:44:22.113878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.113865 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8380d0c7-517d-48a2-9ad7-e5951a060780" containerName="revoke-keys" Nov 03 22:44:22.114006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.113973 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99c0a29-be2a-4799-841d-ea0eaeadbde9" containerName="revoke-keys" Nov 03 22:44:22.135005 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.134950 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.151021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.150582 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn"] Nov 03 22:44:22.151021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.150709 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.181033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.152938 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs94b\" (UniqueName: \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.389113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.371202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.389113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.371357 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.389113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.371627 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.389113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.156094 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.389113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.360644 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:22.466033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.397083 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:44:22.466033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.397397 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.466033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.397536 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:44:22.466033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.409118 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:44:22.606791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.542719 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs94b\" (UniqueName: \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.606791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.542859 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.606791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.543009 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.617016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.609213 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.617016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.609302 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:44:22.617016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.609367 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:44:22.680076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.653553 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.680076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.654300 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:44:22.684504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.681828 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:44:22.684504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.682223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:44:22.709752 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.709718 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15eb386-f08d-4016-8966-b3611ed42380" path="/var/lib/kubelet/pods/a15eb386-f08d-4016-8966-b3611ed42380/volumes" Nov 03 22:44:22.710505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.710484 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:44:22.710801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.710787 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:44:22.750027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.732224 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:44:22.750027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.732284 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:44:22.750027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.732320 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:44:22.775296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.775233 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.887465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.834976 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs94b\" (UniqueName: \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\") pod \"maas-api-revoke-sub-d158bf80-tgcvn\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:22.887465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.849231 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.887465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.850553 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:44:22.960361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:22.960275 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:23.042026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.036845 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:44:23.081789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.073162 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:44:23.101082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.094647 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:44:23.101082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.095704 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:44:23.167016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.137520 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:44:23.167016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.137951 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:44:23.550018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.548368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:44:23.596047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.576320 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:23.613066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.609908 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:23.642494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.632944 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:23.723193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.723144 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 22:44:23.762015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.726342 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 03 22:44:23.843384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.735728 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 22:44:23.843384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.828073 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:44:23.843384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.828237 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:44:24.044015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.040087 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.124842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:23.761415 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.180649 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.180760 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.180887 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.181582 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.181654 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.190772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.181937 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:44:24.227040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.206461 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:44:24.232597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.231387 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/health\": context deadline exceeded" Nov 03 22:44:24.265537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.253442 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:44:24.265537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.255763 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:44:24.378605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.292480 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:44:24.380151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.380127 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v"] Nov 03 22:44:24.380400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.380384 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:44:24.380577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.380556 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:24.380728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.380702 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.425015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.418592 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae82e407-ztm2v"] Nov 03 22:44:24.530022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.497949 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0216f55-629c-4ecf-a004-92cb34f3d040" path="/var/lib/kubelet/pods/f0216f55-629c-4ecf-a004-92cb34f3d040/volumes" Nov 03 22:44:24.582656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.582628 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:44:24.582894 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.582877 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:44:24.594018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.548516 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.606096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.587106 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:44:24.651059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.630813 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 22:44:24.651059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.630597 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.651059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.633668 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:24.844152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.843851 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6"] Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872511 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872582 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872615 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872659 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872678 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6"] Nov 03 22:44:24.875073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.872739 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:44:24.880143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.880115 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:44:24.881493 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.881468 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:44:24.882620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.882604 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:24.883358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.883128 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:44:24.889181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.888859 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:44:24.889181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.889037 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" containerID="cri-o://90fd2b46f9f08d45c733d59b8f503ac25df2c56b99c0ecf8d6b5a4e995589baf" gracePeriod=30 Nov 03 22:44:24.901898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.893438 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:44:24.901898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.898634 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:44:24.901898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.898796 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" containerID="cri-o://bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c" gracePeriod=30 Nov 03 22:44:24.905662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:24.905604 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn"] Nov 03 22:44:25.030270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.030228 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.030919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.030891 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.031076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.030952 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbpb\" (UniqueName: \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.132358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.132323 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.132530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.132367 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbpb\" (UniqueName: \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.132530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.132465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.133126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.133102 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.134899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.134874 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.193353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.193288 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbpb\" (UniqueName: \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\") pod \"maas-api-revoke-sub-c9833a00-tgqg6\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.229150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.229116 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:25.289681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.289182 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" event={"ID":"511cc975-ebbb-47a6-ba87-fd04c4eaacd0","Type":"ContainerStarted","Data":"67f66eda0018e6a57be5af016a27876594a98da5dd9f185f8f354da02aaf238f"} Nov 03 22:44:25.631827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:25.631802 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6"] Nov 03 22:44:25.634378 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:25.634350 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15ffb23_511d_403a_a0bc_a62e1a3662c3.slice/crio-5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9 WatchSource:0}: Error finding container 5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9: Status 404 returned error can't find the container with id 5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9 Nov 03 22:44:26.300800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:26.300770 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" event={"ID":"f15ffb23-511d-403a-a0bc-a62e1a3662c3","Type":"ContainerStarted","Data":"560ee820cc373b414107b8587f3c30685e2c689665db7e1c46733a3203fbbd71"} Nov 03 22:44:26.301275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:26.300829 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" event={"ID":"f15ffb23-511d-403a-a0bc-a62e1a3662c3","Type":"ContainerStarted","Data":"5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9"} Nov 03 22:44:26.302726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:26.302699 5051 generic.go:356] "Generic (PLEG): container finished" podID="511cc975-ebbb-47a6-ba87-fd04c4eaacd0" containerID="cff389e2e1e7d532ac9852ba464d5353566c85f5541994baae4388f4bb90c694" exitCode=0 Nov 03 22:44:26.302864 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:26.302843 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" event={"ID":"511cc975-ebbb-47a6-ba87-fd04c4eaacd0","Type":"ContainerDied","Data":"cff389e2e1e7d532ac9852ba464d5353566c85f5541994baae4388f4bb90c694"} Nov 03 22:44:27.172933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.172865 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" podStartSLOduration=4.172847338 podStartE2EDuration="4.172847338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:27.170231313 +0000 UTC m=+1340.907575541" watchObservedRunningTime="2026-11-03 22:44:27.172847338 +0000 UTC m=+1340.910191563" Nov 03 22:44:27.253937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.253905 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj"] Nov 03 22:44:27.258472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.258455 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.316435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.316389 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj"] Nov 03 22:44:27.317176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.317143 5051 generic.go:356] "Generic (PLEG): container finished" podID="f15ffb23-511d-403a-a0bc-a62e1a3662c3" containerID="560ee820cc373b414107b8587f3c30685e2c689665db7e1c46733a3203fbbd71" exitCode=0 Nov 03 22:44:27.317509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.317483 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" event={"ID":"f15ffb23-511d-403a-a0bc-a62e1a3662c3","Type":"ContainerDied","Data":"560ee820cc373b414107b8587f3c30685e2c689665db7e1c46733a3203fbbd71"} Nov 03 22:44:27.367613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.366840 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76hc\" (UniqueName: \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.367613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.367033 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.367613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.367153 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.469046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.468865 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.469771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.469744 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76hc\" (UniqueName: \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.470181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.470006 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.470725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.470704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.471973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.471932 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.495700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.495678 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:27.529896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.529878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76hc\" (UniqueName: \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\") pod \"maas-api-revoke-sub-f6321b11-rp4sj\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.575018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.574982 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:27.672480 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.672386 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\") pod \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " Nov 03 22:44:27.672612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.672512 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\") pod \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " Nov 03 22:44:27.672720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.672699 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\" (UniqueName: \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\") pod \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\" (UID: \"511cc975-ebbb-47a6-ba87-fd04c4eaacd0\") " Nov 03 22:44:27.672927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.672898 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca" pod "511cc975-ebbb-47a6-ba87-fd04c4eaacd0" (UID: "511cc975-ebbb-47a6-ba87-fd04c4eaacd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:27.673497 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.673471 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:27.676194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.676171 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b" pod "511cc975-ebbb-47a6-ba87-fd04c4eaacd0" (UID: "511cc975-ebbb-47a6-ba87-fd04c4eaacd0"). InnerVolumeSpecName "kube-api-access-rs94b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:27.676273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.676210 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca" pod "511cc975-ebbb-47a6-ba87-fd04c4eaacd0" (UID: "511cc975-ebbb-47a6-ba87-fd04c4eaacd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:27.775309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.775218 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:27.775309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.775258 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs94b\" (UniqueName: \"kubernetes.io/projected/511cc975-ebbb-47a6-ba87-fd04c4eaacd0-kube-api-access-rs94b\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:27.965597 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:27.965562 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa414417_9314_4a0b_b581_78c792f7159a.slice/crio-272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec WatchSource:0}: Error finding container 272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec: Status 404 returned error can't find the container with id 272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec Nov 03 22:44:27.965736 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:27.965674 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj"] Nov 03 22:44:28.326770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.326741 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" event={"ID":"fa414417-9314-4a0b-b581-78c792f7159a","Type":"ContainerStarted","Data":"6e4f531cd42ab83e66d8770fc832f6abd6c743bb506866f4dad8cca92478b6ce"} Nov 03 22:44:28.327218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.326790 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" event={"ID":"fa414417-9314-4a0b-b581-78c792f7159a","Type":"ContainerStarted","Data":"272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec"} Nov 03 22:44:28.328248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.328230 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" Nov 03 22:44:28.328356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.328265 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn" event={"ID":"511cc975-ebbb-47a6-ba87-fd04c4eaacd0","Type":"ContainerDied","Data":"67f66eda0018e6a57be5af016a27876594a98da5dd9f185f8f354da02aaf238f"} Nov 03 22:44:28.328356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.328300 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f66eda0018e6a57be5af016a27876594a98da5dd9f185f8f354da02aaf238f" Nov 03 22:44:28.447545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.447438 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" podStartSLOduration=1.447419421 podStartE2EDuration="1.447419421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:28.441687034 +0000 UTC m=+1342.179031257" watchObservedRunningTime="2026-11-03 22:44:28.447419421 +0000 UTC m=+1342.184763646" Nov 03 22:44:28.491890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.491866 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:28.586308 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.586275 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\") pod \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " Nov 03 22:44:28.586514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.586412 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\") pod \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " Nov 03 22:44:28.586616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.586599 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\" (UniqueName: \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\") pod \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\" (UID: \"f15ffb23-511d-403a-a0bc-a62e1a3662c3\") " Nov 03 22:44:28.587051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.587022 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca" pod "f15ffb23-511d-403a-a0bc-a62e1a3662c3" (UID: "f15ffb23-511d-403a-a0bc-a62e1a3662c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:28.587365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.587342 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:28.589363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.589335 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb" pod "f15ffb23-511d-403a-a0bc-a62e1a3662c3" (UID: "f15ffb23-511d-403a-a0bc-a62e1a3662c3"). InnerVolumeSpecName "kube-api-access-kxbpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:28.589702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.589683 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca" pod "f15ffb23-511d-403a-a0bc-a62e1a3662c3" (UID: "f15ffb23-511d-403a-a0bc-a62e1a3662c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:28.689320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.689288 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbpb\" (UniqueName: \"kubernetes.io/projected/f15ffb23-511d-403a-a0bc-a62e1a3662c3-kube-api-access-kxbpb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:28.689478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:28.689323 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15ffb23-511d-403a-a0bc-a62e1a3662c3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:29.171613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.171580 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2"] Nov 03 22:44:29.172293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172265 5051 cpu_manager.go:414] "containerMap: removing container" podUID="511cc975-ebbb-47a6-ba87-fd04c4eaacd0" containerName="revoke-keys" Nov 03 22:44:29.172293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172285 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511cc975-ebbb-47a6-ba87-fd04c4eaacd0" containerName="revoke-keys" Nov 03 22:44:29.172510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172339 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f15ffb23-511d-403a-a0bc-a62e1a3662c3" containerName="revoke-keys" Nov 03 22:44:29.172510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172347 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15ffb23-511d-403a-a0bc-a62e1a3662c3" containerName="revoke-keys" Nov 03 22:44:29.172602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172536 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="511cc975-ebbb-47a6-ba87-fd04c4eaacd0" containerName="revoke-keys" Nov 03 22:44:29.172602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.172562 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15ffb23-511d-403a-a0bc-a62e1a3662c3" containerName="revoke-keys" Nov 03 22:44:29.176307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.176285 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.196338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.196305 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86j8d\" (UniqueName: \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.196545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.196523 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.196752 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.196733 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.232287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.232213 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2"] Nov 03 22:44:29.298423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.298393 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.298583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.298550 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.298628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.298579 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86j8d\" (UniqueName: \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.299306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.299283 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.300913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.300882 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.339633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.339610 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" Nov 03 22:44:29.341389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.339641 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6" event={"ID":"f15ffb23-511d-403a-a0bc-a62e1a3662c3","Type":"ContainerDied","Data":"5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9"} Nov 03 22:44:29.341389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.339693 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f158dde430dcace0b0b40af9d864cb1cdca0b77ffcab28283952bc660899ec9" Nov 03 22:44:29.342687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.342661 5051 generic.go:356] "Generic (PLEG): container finished" podID="fa414417-9314-4a0b-b581-78c792f7159a" containerID="6e4f531cd42ab83e66d8770fc832f6abd6c743bb506866f4dad8cca92478b6ce" exitCode=0 Nov 03 22:44:29.342846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.342831 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" event={"ID":"fa414417-9314-4a0b-b581-78c792f7159a","Type":"ContainerDied","Data":"6e4f531cd42ab83e66d8770fc832f6abd6c743bb506866f4dad8cca92478b6ce"} Nov 03 22:44:29.359576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.359553 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86j8d\" (UniqueName: \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\") pod \"maas-api-revoke-sub-64093830-j8nm2\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.493477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.493391 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:29.879299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:29.879270 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2"] Nov 03 22:44:29.880242 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:29.880216 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90985c4a_5bac_4e38_bf82_fa667dcf5e86.slice/crio-385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d WatchSource:0}: Error finding container 385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d: Status 404 returned error can't find the container with id 385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d Nov 03 22:44:30.054884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.054820 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9"] Nov 03 22:44:30.062124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.062092 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1a0bb9-6wtc9"] Nov 03 22:44:30.271029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.269515 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:44:30.315684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315649 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.315875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315708 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.315875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315784 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.315875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315860 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315903 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.315980 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.316149 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.316199 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316282 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.316259 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\") pod \"90c6b8c6-7653-475a-817a-1fd48b6013af\" (UID: \"90c6b8c6-7653-475a-817a-1fd48b6013af\") " Nov 03 22:44:30.316813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.316578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.318015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.317164 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.318015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.317579 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.318015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.317743 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.318904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.318872 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.321763 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.321729 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.330369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.330334 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.344187 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.343471 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "kube-api-access-qdqfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.344655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.344630 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.345584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.345561 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy" pod "90c6b8c6-7653-475a-817a-1fd48b6013af" (UID: "90c6b8c6-7653-475a-817a-1fd48b6013af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.357349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.357325 5051 generic.go:356] "Generic (PLEG): container finished" podID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerID="90fd2b46f9f08d45c733d59b8f503ac25df2c56b99c0ecf8d6b5a4e995589baf" exitCode=0 Nov 03 22:44:30.357486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.357465 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" event={"ID":"88b4ba0b-c12e-4ae9-9baa-b801e810544f","Type":"ContainerDied","Data":"90fd2b46f9f08d45c733d59b8f503ac25df2c56b99c0ecf8d6b5a4e995589baf"} Nov 03 22:44:30.359242 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.359216 5051 generic.go:356] "Generic (PLEG): container finished" podID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerID="bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c" exitCode=0 Nov 03 22:44:30.359343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.359280 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" Nov 03 22:44:30.359343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.359320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" event={"ID":"90c6b8c6-7653-475a-817a-1fd48b6013af","Type":"ContainerDied","Data":"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c"} Nov 03 22:44:30.359454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.359358 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5" event={"ID":"90c6b8c6-7653-475a-817a-1fd48b6013af","Type":"ContainerDied","Data":"56358af86d1e7f097797ecf9bd5ae81a7e8cea5fcdc322576116356c833e53fd"} Nov 03 22:44:30.359454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.359381 5051 scope.go:122] "RemoveContainer" containerID="bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c" Nov 03 22:44:30.363091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.363066 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" event={"ID":"90985c4a-5bac-4e38-bf82-fa667dcf5e86","Type":"ContainerStarted","Data":"f45fcb00bb04e3397c813f88cde1fcff0a99bbe70101bf1322fe4167fdeb160f"} Nov 03 22:44:30.363191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.363113 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" event={"ID":"90985c4a-5bac-4e38-bf82-fa667dcf5e86","Type":"ContainerStarted","Data":"385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d"} Nov 03 22:44:30.364075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.364057 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:44:30.372366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.372349 5051 scope.go:122] "RemoveContainer" containerID="bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c" Nov 03 22:44:30.372647 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:44:30.372590 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c\": container with ID starting with bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c not found: ID does not exist" containerID="bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c" Nov 03 22:44:30.372647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.372622 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c"} err="failed to get container status \"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c\": rpc error: code = NotFound desc = could not find container \"bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c\": container with ID starting with bd20f587d953d8d9d10fa4d0dac6d657661846277a2c8cb7e2fa11642e2aa31c not found: ID does not exist" Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.418512 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.418563 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.418617 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.418806 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.418914 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.419024 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.419124 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.419190 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8287ed12-7127-4433-bf00-96e28dac4ee5" path="/var/lib/kubelet/pods/8287ed12-7127-4433-bf00-96e28dac4ee5/volumes" Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.419259 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.419564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.419365 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\") pod \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\" (UID: \"88b4ba0b-c12e-4ae9-9baa-b801e810544f\") " Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420290 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420314 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420332 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqfl\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-kube-api-access-qdqfl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420346 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420359 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90c6b8c6-7653-475a-817a-1fd48b6013af-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420372 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420385 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420398 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90c6b8c6-7653-475a-817a-1fd48b6013af-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420396 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420665 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420901 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.421055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.420941 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.424455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.424427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "kube-api-access-fr2w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.424581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.424479 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.424581 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.424570 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.425951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.425927 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token" pod "88b4ba0b-c12e-4ae9-9baa-b801e810544f" (UID: "88b4ba0b-c12e-4ae9-9baa-b801e810544f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.522389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.520793 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" podStartSLOduration=1.5207790669999999 podStartE2EDuration="1.520779067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:30.514383534 +0000 UTC m=+1344.251727757" watchObservedRunningTime="2026-11-03 22:44:30.520779067 +0000 UTC m=+1344.258123290" Nov 03 22:44:30.522771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522748 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.522771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522772 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522782 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522790 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522799 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522809 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522818 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522825 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b4ba0b-c12e-4ae9-9baa-b801e810544f-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.523079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.522834 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2w4\" (UniqueName: \"kubernetes.io/projected/88b4ba0b-c12e-4ae9-9baa-b801e810544f-kube-api-access-fr2w4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.550229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.550207 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:30.623388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.623352 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\") pod \"fa414417-9314-4a0b-b581-78c792f7159a\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " Nov 03 22:44:30.623655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.623633 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\") pod \"fa414417-9314-4a0b-b581-78c792f7159a\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " Nov 03 22:44:30.623778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.623751 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\" (UniqueName: \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\") pod \"fa414417-9314-4a0b-b581-78c792f7159a\" (UID: \"fa414417-9314-4a0b-b581-78c792f7159a\") " Nov 03 22:44:30.624662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.624639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca" pod "fa414417-9314-4a0b-b581-78c792f7159a" (UID: "fa414417-9314-4a0b-b581-78c792f7159a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.626775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.626749 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc" pod "fa414417-9314-4a0b-b581-78c792f7159a" (UID: "fa414417-9314-4a0b-b581-78c792f7159a"). InnerVolumeSpecName "kube-api-access-b76hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.626925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.626904 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca" pod "fa414417-9314-4a0b-b581-78c792f7159a" (UID: "fa414417-9314-4a0b-b581-78c792f7159a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:30.629808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.629785 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:44:30.647825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.639079 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2jqg5"] Nov 03 22:44:30.725414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.725385 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.725414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.725414 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa414417-9314-4a0b-b581-78c792f7159a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.725594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:30.725424 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76hc\" (UniqueName: \"kubernetes.io/projected/fa414417-9314-4a0b-b581-78c792f7159a-kube-api-access-b76hc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:31.373783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.373757 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" Nov 03 22:44:31.373783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.373770 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv" event={"ID":"88b4ba0b-c12e-4ae9-9baa-b801e810544f","Type":"ContainerDied","Data":"9372deef244010f575ec8d247a471b2f134cdf1388b05dec08ad9fb6b0a23f76"} Nov 03 22:44:31.374341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.373860 5051 scope.go:122] "RemoveContainer" containerID="90fd2b46f9f08d45c733d59b8f503ac25df2c56b99c0ecf8d6b5a4e995589baf" Nov 03 22:44:31.375653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.375627 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" event={"ID":"fa414417-9314-4a0b-b581-78c792f7159a","Type":"ContainerDied","Data":"272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec"} Nov 03 22:44:31.375653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.375644 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj" Nov 03 22:44:31.375813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.375672 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272aabaa322842612e67c720e2d1473016aa869dc429ef1d46cdfdb831aa03ec" Nov 03 22:44:31.385929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.385905 5051 generic.go:356] "Generic (PLEG): container finished" podID="90985c4a-5bac-4e38-bf82-fa667dcf5e86" containerID="f45fcb00bb04e3397c813f88cde1fcff0a99bbe70101bf1322fe4167fdeb160f" exitCode=0 Nov 03 22:44:31.386087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.386029 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" event={"ID":"90985c4a-5bac-4e38-bf82-fa667dcf5e86","Type":"ContainerDied","Data":"f45fcb00bb04e3397c813f88cde1fcff0a99bbe70101bf1322fe4167fdeb160f"} Nov 03 22:44:31.544520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.544491 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:44:31.549187 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:31.549168 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hn9sv"] Nov 03 22:44:32.410828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.410789 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" path="/var/lib/kubelet/pods/88b4ba0b-c12e-4ae9-9baa-b801e810544f/volumes" Nov 03 22:44:32.411375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.411357 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" path="/var/lib/kubelet/pods/90c6b8c6-7653-475a-817a-1fd48b6013af/volumes" Nov 03 22:44:32.565135 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.565109 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:32.652984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.652915 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\") pod \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " Nov 03 22:44:32.652984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.652956 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\") pod \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " Nov 03 22:44:32.653230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.653009 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\" (UniqueName: \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\") pod \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\" (UID: \"90985c4a-5bac-4e38-bf82-fa667dcf5e86\") " Nov 03 22:44:32.653290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.653240 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca" pod "90985c4a-5bac-4e38-bf82-fa667dcf5e86" (UID: "90985c4a-5bac-4e38-bf82-fa667dcf5e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:32.653572 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.653555 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:32.655308 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.655289 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca" pod "90985c4a-5bac-4e38-bf82-fa667dcf5e86" (UID: "90985c4a-5bac-4e38-bf82-fa667dcf5e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:32.655406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.655313 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d" pod "90985c4a-5bac-4e38-bf82-fa667dcf5e86" (UID: "90985c4a-5bac-4e38-bf82-fa667dcf5e86"). InnerVolumeSpecName "kube-api-access-86j8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:32.755047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.755014 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90985c4a-5bac-4e38-bf82-fa667dcf5e86-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:32.755047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:32.755044 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86j8d\" (UniqueName: \"kubernetes.io/projected/90985c4a-5bac-4e38-bf82-fa667dcf5e86-kube-api-access-86j8d\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:33.407171 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:33.407143 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" Nov 03 22:44:33.407342 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:33.407200 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2" event={"ID":"90985c4a-5bac-4e38-bf82-fa667dcf5e86","Type":"ContainerDied","Data":"385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d"} Nov 03 22:44:33.407342 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:33.407248 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="385f1682f494678d7e81c0a9b419db5e699587c58b95bcd0215447de4b71cc3d" Nov 03 22:44:37.912896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.912863 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8"] Nov 03 22:44:37.913537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913521 5051 cpu_manager.go:414] "containerMap: removing container" podUID="90985c4a-5bac-4e38-bf82-fa667dcf5e86" containerName="revoke-keys" Nov 03 22:44:37.913578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913538 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90985c4a-5bac-4e38-bf82-fa667dcf5e86" containerName="revoke-keys" Nov 03 22:44:37.913578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913557 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fa414417-9314-4a0b-b581-78c792f7159a" containerName="revoke-keys" Nov 03 22:44:37.913578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913561 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa414417-9314-4a0b-b581-78c792f7159a" containerName="revoke-keys" Nov 03 22:44:37.913711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913597 5051 cpu_manager.go:414] "containerMap: removing container" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" Nov 03 22:44:37.913711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913607 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" Nov 03 22:44:37.913711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913620 5051 cpu_manager.go:414] "containerMap: removing container" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" Nov 03 22:44:37.913711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913625 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" Nov 03 22:44:37.913823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913722 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c6b8c6-7653-475a-817a-1fd48b6013af" containerName="istio-proxy" Nov 03 22:44:37.913823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913730 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b4ba0b-c12e-4ae9-9baa-b801e810544f" containerName="istio-proxy" Nov 03 22:44:37.913823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913740 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="90985c4a-5bac-4e38-bf82-fa667dcf5e86" containerName="revoke-keys" Nov 03 22:44:37.913823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.913748 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa414417-9314-4a0b-b581-78c792f7159a" containerName="revoke-keys" Nov 03 22:44:37.917536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.917517 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:37.977378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:37.977345 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8"] Nov 03 22:44:38.018165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.018135 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:38.018366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.018286 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:44:38.028702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.028675 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.028796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.028723 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ztk\" (UniqueName: \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.028855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.028814 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.130397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.130367 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.130397 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.130418 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ztk\" (UniqueName: \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.130624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.130476 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.130983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.130954 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.133200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.133179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.192127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.192065 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ztk\" (UniqueName: \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\") pod \"maas-api-revoke-sub-e34ad0d5-nx5f8\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.231731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.231702 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:38.618583 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:38.618542 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8"] Nov 03 22:44:38.627174 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:38.627143 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7a1e3f_2e64_432d_8a56_d51db575b5cd.slice/crio-4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e WatchSource:0}: Error finding container 4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e: Status 404 returned error can't find the container with id 4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e Nov 03 22:44:39.308465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.308394 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v"] Nov 03 22:44:39.312924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.312906 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.370975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.370945 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v"] Nov 03 22:44:39.451661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.451627 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfcn\" (UniqueName: \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.451843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.451706 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.451843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.451823 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.458614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.458579 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" event={"ID":"0e7a1e3f-2e64-432d-8a56-d51db575b5cd","Type":"ContainerStarted","Data":"c6aa075ab131853e61674c21e4303fa70789a57f88e6b4003f1b725fc60ca018"} Nov 03 22:44:39.458735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.458647 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" event={"ID":"0e7a1e3f-2e64-432d-8a56-d51db575b5cd","Type":"ContainerStarted","Data":"4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e"} Nov 03 22:44:39.553378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.553350 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.553546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.553531 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfcn\" (UniqueName: \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.553612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.553570 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.554218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.554196 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.555801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.555782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.576354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.576307 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" podStartSLOduration=2.576290772 podStartE2EDuration="2.576290772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:39.573180509 +0000 UTC m=+1353.310524732" watchObservedRunningTime="2026-11-03 22:44:39.576290772 +0000 UTC m=+1353.313634997" Nov 03 22:44:39.614719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.614694 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfcn\" (UniqueName: \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\") pod \"maas-api-revoke-sub-33566b61-xkr9v\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.627787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.627769 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:39.911185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.911155 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl"] Nov 03 22:44:39.916777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.916755 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:39.974961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:39.974934 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl"] Nov 03 22:44:40.028380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.028355 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v"] Nov 03 22:44:40.032131 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:40.032106 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d7bdc_c22b_4c3f_be8e_66331f5f8320.slice/crio-628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7 WatchSource:0}: Error finding container 628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7: Status 404 returned error can't find the container with id 628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7 Nov 03 22:44:40.050980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.050957 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:44:40.056919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.056895 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.060538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.060510 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.060686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.060667 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.060745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.060694 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79nb\" (UniqueName: \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165330 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165404 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165499 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165557 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165656 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165713 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165739 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79nb\" (UniqueName: \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165768 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89h7z\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165886 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.165909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.169197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.167735 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.227044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.185420 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-468d8d1b-openshift-default-dockercfg-xhvjp\"" Nov 03 22:44:40.227044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.194482 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:44:40.227044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.195096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.241019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.240935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79nb\" (UniqueName: \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\") pod \"maas-api-revoke-sub-76e5764a-zqhrl\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.268058 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268024 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89h7z\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268083 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268115 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268160 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268232 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268401 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268459 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268579 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.268698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.268681 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.272625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.269123 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.272625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.269574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.272625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.269844 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.272625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.270569 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.272625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.271799 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.297061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.282312 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.297061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.282580 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.331705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.331674 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.334188 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.334157 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89h7z\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\") pod \"e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.410173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.406861 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:40.491820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.486763 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" event={"ID":"745d7bdc-c22b-4c3f-be8e-66331f5f8320","Type":"ContainerStarted","Data":"628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7"} Nov 03 22:44:40.522414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.508355 5051 generic.go:356] "Generic (PLEG): container finished" podID="0e7a1e3f-2e64-432d-8a56-d51db575b5cd" containerID="c6aa075ab131853e61674c21e4303fa70789a57f88e6b4003f1b725fc60ca018" exitCode=0 Nov 03 22:44:40.522414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.508460 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" event={"ID":"0e7a1e3f-2e64-432d-8a56-d51db575b5cd","Type":"ContainerDied","Data":"c6aa075ab131853e61674c21e4303fa70789a57f88e6b4003f1b725fc60ca018"} Nov 03 22:44:40.531673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.531263 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:40.900195 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:40.900166 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1573b327_16ea_4f19_832b_8559cdfec54b.slice/crio-1dc5076740ff1f03ce401298fa2f962029dd3d42c712de8f25d6c36336356bde WatchSource:0}: Error finding container 1dc5076740ff1f03ce401298fa2f962029dd3d42c712de8f25d6c36336356bde: Status 404 returned error can't find the container with id 1dc5076740ff1f03ce401298fa2f962029dd3d42c712de8f25d6c36336356bde Nov 03 22:44:40.902169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.902134 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:40.902281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.902200 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:40.902281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.902229 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:40.902384 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.902343 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:44:40.953734 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:40.953705 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73544a1d_21b6_4216_8ff2_40b63b12a9bc.slice/crio-21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2 WatchSource:0}: Error finding container 21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2: Status 404 returned error can't find the container with id 21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2 Nov 03 22:44:40.954057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:40.954038 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl"] Nov 03 22:44:41.520209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.520114 5051 generic.go:356] "Generic (PLEG): container finished" podID="745d7bdc-c22b-4c3f-be8e-66331f5f8320" containerID="8989e5ab3d8439a62fa61004225948d39b730f3163fed92b0977d0f3729c748b" exitCode=0 Nov 03 22:44:41.520661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.520204 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" event={"ID":"745d7bdc-c22b-4c3f-be8e-66331f5f8320","Type":"ContainerDied","Data":"8989e5ab3d8439a62fa61004225948d39b730f3163fed92b0977d0f3729c748b"} Nov 03 22:44:41.522173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.522142 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" event={"ID":"73544a1d-21b6-4216-8ff2-40b63b12a9bc","Type":"ContainerStarted","Data":"a59dfb75d1584216cb5ad4c30ea366cf4403f00c8c2a8657c083b12a5d3efcd5"} Nov 03 22:44:41.522317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.522195 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" event={"ID":"73544a1d-21b6-4216-8ff2-40b63b12a9bc","Type":"ContainerStarted","Data":"21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2"} Nov 03 22:44:41.523971 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.523947 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerStarted","Data":"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0"} Nov 03 22:44:41.524095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.524031 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerStarted","Data":"1dc5076740ff1f03ce401298fa2f962029dd3d42c712de8f25d6c36336356bde"} Nov 03 22:44:41.693316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.693296 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:41.760024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.759953 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podStartSLOduration=1.759940692 podStartE2EDuration="1.759940692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:41.756472736 +0000 UTC m=+1355.493816959" watchObservedRunningTime="2026-11-03 22:44:41.759940692 +0000 UTC m=+1355.497284996" Nov 03 22:44:41.800242 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.800181 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\") pod \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " Nov 03 22:44:41.800350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.800286 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\" (UniqueName: \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\") pod \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " Nov 03 22:44:41.800492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.800476 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\") pod \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\" (UID: \"0e7a1e3f-2e64-432d-8a56-d51db575b5cd\") " Nov 03 22:44:41.800779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.800757 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca" pod "0e7a1e3f-2e64-432d-8a56-d51db575b5cd" (UID: "0e7a1e3f-2e64-432d-8a56-d51db575b5cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:41.801125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.801106 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:41.802309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.802287 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca" pod "0e7a1e3f-2e64-432d-8a56-d51db575b5cd" (UID: "0e7a1e3f-2e64-432d-8a56-d51db575b5cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:41.802729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.802710 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk" pod "0e7a1e3f-2e64-432d-8a56-d51db575b5cd" (UID: "0e7a1e3f-2e64-432d-8a56-d51db575b5cd"). InnerVolumeSpecName "kube-api-access-25ztk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:41.893505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.891221 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" podStartSLOduration=2.891203907 podStartE2EDuration="2.891203907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:41.875739193 +0000 UTC m=+1355.613083417" watchObservedRunningTime="2026-11-03 22:44:41.891203907 +0000 UTC m=+1355.628548132" Nov 03 22:44:41.903400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.903367 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:41.903400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:41.903402 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25ztk\" (UniqueName: \"kubernetes.io/projected/0e7a1e3f-2e64-432d-8a56-d51db575b5cd-kube-api-access-25ztk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:42.409757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.409727 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:42.413301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.413277 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:42.536006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.535803 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" event={"ID":"0e7a1e3f-2e64-432d-8a56-d51db575b5cd","Type":"ContainerDied","Data":"4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e"} Nov 03 22:44:42.536006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.535853 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8" Nov 03 22:44:42.536006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.535892 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e8b99e8f66f5a219e908593b08de348b6f62ce07f8cc76caae4940660e89e3e" Nov 03 22:44:42.539575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.539549 5051 generic.go:356] "Generic (PLEG): container finished" podID="73544a1d-21b6-4216-8ff2-40b63b12a9bc" containerID="a59dfb75d1584216cb5ad4c30ea366cf4403f00c8c2a8657c083b12a5d3efcd5" exitCode=0 Nov 03 22:44:42.539716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.539580 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" event={"ID":"73544a1d-21b6-4216-8ff2-40b63b12a9bc","Type":"ContainerDied","Data":"a59dfb75d1584216cb5ad4c30ea366cf4403f00c8c2a8657c083b12a5d3efcd5"} Nov 03 22:44:42.540423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.540008 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:42.540846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.540826 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:44:42.709625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.709600 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:42.815780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.815709 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\" (UniqueName: \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\") pod \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " Nov 03 22:44:42.815780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.815743 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\") pod \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " Nov 03 22:44:42.815944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.815810 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\") pod \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\" (UID: \"745d7bdc-c22b-4c3f-be8e-66331f5f8320\") " Nov 03 22:44:42.816235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.816210 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca" pod "745d7bdc-c22b-4c3f-be8e-66331f5f8320" (UID: "745d7bdc-c22b-4c3f-be8e-66331f5f8320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:42.816547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.816498 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:42.817964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.817942 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca" pod "745d7bdc-c22b-4c3f-be8e-66331f5f8320" (UID: "745d7bdc-c22b-4c3f-be8e-66331f5f8320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:42.818094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.817966 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn" pod "745d7bdc-c22b-4c3f-be8e-66331f5f8320" (UID: "745d7bdc-c22b-4c3f-be8e-66331f5f8320"). InnerVolumeSpecName "kube-api-access-znfcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:42.918008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.917966 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfcn\" (UniqueName: \"kubernetes.io/projected/745d7bdc-c22b-4c3f-be8e-66331f5f8320-kube-api-access-znfcn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:42.918177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:42.918036 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745d7bdc-c22b-4c3f-be8e-66331f5f8320-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:43.550963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.550929 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" Nov 03 22:44:43.551452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.550967 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v" event={"ID":"745d7bdc-c22b-4c3f-be8e-66331f5f8320","Type":"ContainerDied","Data":"628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7"} Nov 03 22:44:43.551452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.551048 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="628f5399022ba9cca615d00f4765ecb4de39603d85d8f92557722a7aade8acd7" Nov 03 22:44:43.719551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.719523 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:43.832079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.832014 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\" (UniqueName: \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\") pod \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " Nov 03 22:44:43.832079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.832053 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\") pod \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " Nov 03 22:44:43.832315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.832188 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\") pod \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\" (UID: \"73544a1d-21b6-4216-8ff2-40b63b12a9bc\") " Nov 03 22:44:43.832575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.832550 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca" pod "73544a1d-21b6-4216-8ff2-40b63b12a9bc" (UID: "73544a1d-21b6-4216-8ff2-40b63b12a9bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:43.832914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.832845 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:43.839138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.839112 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca" pod "73544a1d-21b6-4216-8ff2-40b63b12a9bc" (UID: "73544a1d-21b6-4216-8ff2-40b63b12a9bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:43.839244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.839132 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb" pod "73544a1d-21b6-4216-8ff2-40b63b12a9bc" (UID: "73544a1d-21b6-4216-8ff2-40b63b12a9bc"). InnerVolumeSpecName "kube-api-access-d79nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:43.933798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.933750 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79nb\" (UniqueName: \"kubernetes.io/projected/73544a1d-21b6-4216-8ff2-40b63b12a9bc-kube-api-access-d79nb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:43.933798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:43.933797 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73544a1d-21b6-4216-8ff2-40b63b12a9bc-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:44.560982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:44.560952 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" Nov 03 22:44:44.561381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:44.561003 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl" event={"ID":"73544a1d-21b6-4216-8ff2-40b63b12a9bc","Type":"ContainerDied","Data":"21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2"} Nov 03 22:44:44.561381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:44.561054 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c61f03759fac110b2acffa12cf20ce1e853936cb1ee3f63cab2f9e41aef3f2" Nov 03 22:44:46.050329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:46.050295 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm"] Nov 03 22:44:46.099149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:46.099123 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338b4283-pqggm"] Nov 03 22:44:46.413157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:46.413119 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6b8c88-1459-4e77-8ca2-3ed3eb131507" path="/var/lib/kubelet/pods/2f6b8c88-1459-4e77-8ca2-3ed3eb131507/volumes" Nov 03 22:44:49.470511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.470475 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:44:49.471505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471487 5051 cpu_manager.go:414] "containerMap: removing container" podUID="73544a1d-21b6-4216-8ff2-40b63b12a9bc" containerName="revoke-keys" Nov 03 22:44:49.471549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471509 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73544a1d-21b6-4216-8ff2-40b63b12a9bc" containerName="revoke-keys" Nov 03 22:44:49.471549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471534 5051 cpu_manager.go:414] "containerMap: removing container" podUID="745d7bdc-c22b-4c3f-be8e-66331f5f8320" containerName="revoke-keys" Nov 03 22:44:49.471549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471542 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745d7bdc-c22b-4c3f-be8e-66331f5f8320" containerName="revoke-keys" Nov 03 22:44:49.471645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471566 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0e7a1e3f-2e64-432d-8a56-d51db575b5cd" containerName="revoke-keys" Nov 03 22:44:49.471645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471574 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7a1e3f-2e64-432d-8a56-d51db575b5cd" containerName="revoke-keys" Nov 03 22:44:49.471739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471728 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="73544a1d-21b6-4216-8ff2-40b63b12a9bc" containerName="revoke-keys" Nov 03 22:44:49.471770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471744 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="745d7bdc-c22b-4c3f-be8e-66331f5f8320" containerName="revoke-keys" Nov 03 22:44:49.471770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.471757 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7a1e3f-2e64-432d-8a56-d51db575b5cd" containerName="revoke-keys" Nov 03 22:44:49.477201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.477173 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.533950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.533923 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:44:49.546340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546306 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546431 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546535 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bxh\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546666 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546650 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546707 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546726 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546817 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546900 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546849 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.546900 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.546864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.648683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.648653 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.648858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.648717 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.648858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.648804 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bxh\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.648858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.648833 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649004 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649048 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649051 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649156 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649214 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649238 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649494 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649567 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.649655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.649567 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.651357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.651337 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.651532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.651515 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.709673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.709642 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.709776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.709693 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bxh\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:49.792286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:49.792207 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:50.203082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.203048 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn"] Nov 03 22:44:50.208711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.208690 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.243438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.243417 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:44:50.253271 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:50.253249 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107e0bc0_43c5_40d0_b73a_b34531285f5d.slice/crio-8c015327a8f07fe77e08bb0e8907d6b1b9b2ecd5f06df21a935ebaf93fba2c45 WatchSource:0}: Error finding container 8c015327a8f07fe77e08bb0e8907d6b1b9b2ecd5f06df21a935ebaf93fba2c45: Status 404 returned error can't find the container with id 8c015327a8f07fe77e08bb0e8907d6b1b9b2ecd5f06df21a935ebaf93fba2c45 Nov 03 22:44:50.255688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.255661 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:50.255794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.255724 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:50.255794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.255754 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:50.258574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.258548 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.258672 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.258592 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wh44\" (UniqueName: \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.259551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.258787 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.263661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.263640 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:50.267772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.265541 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn"] Nov 03 22:44:50.288260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.287404 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:44:50.360726 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.360700 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.360819 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.360744 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wh44\" (UniqueName: \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.360885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.360834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.361511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.361486 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.363151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.363130 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.421784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.421752 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wh44\" (UniqueName: \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\") pod \"maas-api-revoke-sub-b9f62432-cmsdn\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.524299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.524227 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:50.716010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.698943 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" event={"ID":"107e0bc0-43c5-40d0-b73a-b34531285f5d","Type":"ContainerStarted","Data":"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3"} Nov 03 22:44:50.716010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.699087 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" event={"ID":"107e0bc0-43c5-40d0-b73a-b34531285f5d","Type":"ContainerStarted","Data":"8c015327a8f07fe77e08bb0e8907d6b1b9b2ecd5f06df21a935ebaf93fba2c45"} Nov 03 22:44:50.792808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.792726 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:44:50.827952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.827904 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podStartSLOduration=1.8278861960000001 podStartE2EDuration="1.827886196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:50.825849128 +0000 UTC m=+1364.563193373" watchObservedRunningTime="2026-11-03 22:44:50.827886196 +0000 UTC m=+1364.565230419" Nov 03 22:44:50.924713 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:44:50.924689 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fce95cc_b0a4_4232_ad93_380c0627a425.slice/crio-13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7 WatchSource:0}: Error finding container 13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7: Status 404 returned error can't find the container with id 13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7 Nov 03 22:44:50.926176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:50.926151 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn"] Nov 03 22:44:51.627597 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:44:51.627565 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:44:51.709526 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:51.709486 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" event={"ID":"2fce95cc-b0a4-4232-ad93-380c0627a425","Type":"ContainerStarted","Data":"161c4f8856886f66ff3f4a0ff3677ed4ce6d2f89c25aabdade906211f126f543"} Nov 03 22:44:51.709695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:51.709552 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" event={"ID":"2fce95cc-b0a4-4232-ad93-380c0627a425","Type":"ContainerStarted","Data":"13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7"} Nov 03 22:44:51.795113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:51.795081 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:51.795286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:51.795148 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:51.828468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:51.828421 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" podStartSLOduration=1.828406709 podStartE2EDuration="1.828406709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:51.826386195 +0000 UTC m=+1365.563730424" watchObservedRunningTime="2026-11-03 22:44:51.828406709 +0000 UTC m=+1365.565750974" Nov 03 22:44:52.719778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:52.719742 5051 generic.go:356] "Generic (PLEG): container finished" podID="2fce95cc-b0a4-4232-ad93-380c0627a425" containerID="161c4f8856886f66ff3f4a0ff3677ed4ce6d2f89c25aabdade906211f126f543" exitCode=0 Nov 03 22:44:52.720339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:52.719819 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" event={"ID":"2fce95cc-b0a4-4232-ad93-380c0627a425","Type":"ContainerDied","Data":"161c4f8856886f66ff3f4a0ff3677ed4ce6d2f89c25aabdade906211f126f543"} Nov 03 22:44:52.792802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:52.792770 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:52.793113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:52.793086 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:53.793537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:53.793506 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:53.794021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:53.793974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:53.964486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:53.964151 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:54.032669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.032594 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\") pod \"2fce95cc-b0a4-4232-ad93-380c0627a425\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " Nov 03 22:44:54.032809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.032695 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\" (UniqueName: \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\") pod \"2fce95cc-b0a4-4232-ad93-380c0627a425\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " Nov 03 22:44:54.032876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.032861 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\") pod \"2fce95cc-b0a4-4232-ad93-380c0627a425\" (UID: \"2fce95cc-b0a4-4232-ad93-380c0627a425\") " Nov 03 22:44:54.032945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.032921 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca" pod "2fce95cc-b0a4-4232-ad93-380c0627a425" (UID: "2fce95cc-b0a4-4232-ad93-380c0627a425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:54.033436 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.033407 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.034943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.034920 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca" pod "2fce95cc-b0a4-4232-ad93-380c0627a425" (UID: "2fce95cc-b0a4-4232-ad93-380c0627a425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:54.034943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.034935 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44" pod "2fce95cc-b0a4-4232-ad93-380c0627a425" (UID: "2fce95cc-b0a4-4232-ad93-380c0627a425"). InnerVolumeSpecName "kube-api-access-6wh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:54.135031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.134981 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fce95cc-b0a4-4232-ad93-380c0627a425-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.135031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.135029 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wh44\" (UniqueName: \"kubernetes.io/projected/2fce95cc-b0a4-4232-ad93-380c0627a425-kube-api-access-6wh44\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:54.794149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.794051 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:54.794898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.794860 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:54.815694 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.815666 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" event={"ID":"2fce95cc-b0a4-4232-ad93-380c0627a425","Type":"ContainerDied","Data":"13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7"} Nov 03 22:44:54.816600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.816567 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b8909cb6ab94f8c8b6a5bfa53397f3aa9877abb4d8fa0551828de2e14ef5c7" Nov 03 22:44:54.816808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:54.816753 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn" Nov 03 22:44:55.441362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.441315 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.473259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.473219 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.581927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.581141 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.615198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.615070 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.793301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.793211 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:55.793577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.793549 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:55.826025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.822872 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:44:55.847626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.847581 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.900126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.900084 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.901816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.900353 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:44:55.934016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.928088 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:55.934016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:55.928457 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:44:56.211015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.210600 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.279148 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.279100 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.349258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.349210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.476760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.476544 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.478082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.478055 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:44:56.599809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.599756 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.600428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.600389 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:44:56.724013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.721278 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:44:56.724013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.721626 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:44:56.815072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.782684 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:44:56.815072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.801340 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:56.815072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.801396 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:56.868041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.855356 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:44:56.868041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.855737 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:44:56.912120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.912076 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.912387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.912362 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.912608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.912589 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:44:56.912671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.912647 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:44:56.956279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.956240 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.018027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.961690 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:44:57.018027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.961972 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:44:57.018027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:56.962051 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:44:57.241015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.234247 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.242227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.242200 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:44:57.291547 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.291494 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:44:57.292100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.292073 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:44:57.494654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.494538 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:44:57.559400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.559360 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.632017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.621098 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:44:57.730100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.729959 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.846835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.731110 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.846835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.732140 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:44:57.846835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.815974 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:57.846835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.816049 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:57.855665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.855629 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.935442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.931605 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.935442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.932005 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:44:57.971659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.947373 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.989288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.982126 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:44:57.991203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:57.991166 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:44:58.166568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.166526 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.186154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.186114 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.242847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.242811 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:44:58.292038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.291980 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:44:58.626692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.414796 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:44:58.626692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.462243 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.626692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.573085 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.627584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.627120 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.627584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.627498 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.652735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.652137 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:58.652735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.652197 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.965081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.964618 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:44:58.966024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:58.964980 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:44:59.000381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.000319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.098190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.098140 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.098793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.098764 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 03 22:44:59.154098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.101169 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.155284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.102301 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:59.202015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.192091 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.381015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.126849 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:44:59.381015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.126767 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.381015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.127405 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:44:59.381015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.362589 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:44:59.381015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.364664 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.617968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.600247 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:59.788278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.788181 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:44:59.788771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.788742 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:44:59.816641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.815075 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:44:59.816641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.815780 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:59.816641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.815822 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.816641 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.815877 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.818516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.817212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.846307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.846271 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:45:00.128190 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.128158 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:45:00.128746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.073110 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.138019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:44:59.847239 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:45:00.138485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.085749 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:00.138675 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.138651 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.138786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.120405 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.146235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.073164 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:45:00.146641 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:00.086724 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.68s" Nov 03 22:45:00.146815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.146800 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:45:00.152597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.118530 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:45:00.152597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.127935 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.158739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.158716 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:45:00.220714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.200076 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.368014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.355473 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.270121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.369066 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.279021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.369126 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:45:01.297200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.840655 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.297777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.840690 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.298197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.840713 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:45:01.302010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.840749 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:45:01.328402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.883159 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:01.328703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.328676 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.328844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.154078 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.329195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.154117 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.329572 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.154412 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.336017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.165499 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.345013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.166490 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.358010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.296622 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.427636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.129080 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.567265 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.892073 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:01.567543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.567494 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.669676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.669644 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:01.671134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.671104 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2fce95cc-b0a4-4232-ad93-380c0627a425" containerName="revoke-keys" Nov 03 22:45:01.694448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.694419 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fce95cc-b0a4-4232-ad93-380c0627a425" containerName="revoke-keys" Nov 03 22:45:01.695116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.694984 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fce95cc-b0a4-4232-ad93-380c0627a425" containerName="revoke-keys" Nov 03 22:45:01.720550 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.694123 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.847072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:00.909222 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:45:01.847072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.835149 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:45:01.888336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.881165 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:45:01.888336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.881248 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:45:02.000056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:01.997982 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:02.021937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.021897 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.052083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.048582 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.128445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.128412 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:45:02.220047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.213558 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.280902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.280861 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4fp\" (UniqueName: \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.281646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.281618 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.281907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.281885 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97gm\" (UniqueName: \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.292336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.280237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:45:02.318273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.280196 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.318545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.318501 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.318834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.318803 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.323954 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.323924 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:45:02.329516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.328537 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 03 22:45:02.330741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.329711 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:45:02.330741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.329802 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.331899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.330961 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 03 22:45:02.337062 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.336440 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.375132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.351188 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.430042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.429907 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:02.430462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.430392 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.430462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.430443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4fp\" (UniqueName: \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.430663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.430602 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.430746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.430663 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97gm\" (UniqueName: \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.431934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.431902 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.433081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.433059 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.550732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.550702 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4fp\" (UniqueName: \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.559265 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.559222 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:02.565588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.565419 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97gm\" (UniqueName: \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\") pod \"maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.566554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.566202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": context deadline exceeded" Nov 03 22:45:02.586519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.586497 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.640289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.640263 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:02.648037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.645094 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglcl\" (UniqueName: \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.648536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.648491 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.720142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.720054 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:02.757901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.757165 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.757901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.757414 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglcl\" (UniqueName: \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.760067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.759073 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:02.798008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.795221 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:02.798008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.796419 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:02.815787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.815760 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 22:45:02.914020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.909509 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 22:45:02.914020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.832757 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 22:45:02.965038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.962669 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbl55\" (UniqueName: \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:02.965038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.962729 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:02.972671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.970278 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:45:02.981824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.978728 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:45:02.981824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.978799 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:45:02.989101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:02.989066 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 22:45:03.007218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.007196 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:03.018931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.016015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglcl\" (UniqueName: \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:03.079979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.078525 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:03.079979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.078630 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:45:03.079979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.078692 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:45:03.079979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.078738 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:45:03.079979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.079730 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.096326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.096298 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbl55\" (UniqueName: \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:03.096554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.096533 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:03.104981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.100533 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:03.104981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.103457 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 22:45:03.118419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.118396 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 22:45:03.146312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.144180 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:03.146312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.145062 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:45:03.175449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.175421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbl55\" (UniqueName: \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\") pod \"maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:03.184889 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.184167 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:45:03.201553 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.201529 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:03.202700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.202659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nxh\" (UniqueName: \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.203350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.203327 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.224584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.224273 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:03.256280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.251923 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:03.282649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.277475 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:45:03.305359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.305332 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:03.308528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.306785 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.308528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.306967 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nxh\" (UniqueName: \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.308788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.308769 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.376378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.370668 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nxh\" (UniqueName: \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.416707 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.416685 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:03.441049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.440637 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:03.454156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.453928 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:03.513049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.512836 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:03.816133 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.816103 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:03.999417 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:03.999388 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:04.004500 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:04.004467 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad2bb11_a90c_4670_b3bf_9e2645b4ff42.slice/crio-cd3f0cb93bdbafd89b941f0bc7731aa716072f71e7956a55f6e11e0489294b38 WatchSource:0}: Error finding container cd3f0cb93bdbafd89b941f0bc7731aa716072f71e7956a55f6e11e0489294b38: Status 404 returned error can't find the container with id cd3f0cb93bdbafd89b941f0bc7731aa716072f71e7956a55f6e11e0489294b38 Nov 03 22:45:04.005482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.005448 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:04.173871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.173838 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerStarted","Data":"61619e2ee2318520ce226e6a6ba4994083f2fe62b2dcf952d931ea4910c758a5"} Nov 03 22:45:04.177273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.177243 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerStarted","Data":"425f39a8eded6371c4170c50882cfcd8934a71bb20a8db9beb5feb8af4f41529"} Nov 03 22:45:04.177412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.177298 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerStarted","Data":"7c400b6c2ba4ac793e2187639fd6cdf849b1207f63aaf2c4bb39e5fdab994cf9"} Nov 03 22:45:04.181763 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.181739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerStarted","Data":"e3e908dcfa8eb345e238d29f740fde7d02e2568c9530b5168ebc4860e61863d4"} Nov 03 22:45:04.184177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.184150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerStarted","Data":"cd3f0cb93bdbafd89b941f0bc7731aa716072f71e7956a55f6e11e0489294b38"} Nov 03 22:45:04.186767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.186746 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerStarted","Data":"1d4e8606f24e2291482ed4d710f5f5765e6fea61907e7b89510d5bc09fa32411"} Nov 03 22:45:04.187050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.187025 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerStarted","Data":"c43cf6092d0ec033333f9964bc52d8b22c4b9aad59d9ae9edfb502e2cc1b4552"} Nov 03 22:45:04.794887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.794852 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:45:04.845117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.845085 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:04.845295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:04.845139 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.019916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.019810 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" podStartSLOduration=5.019790946 podStartE2EDuration="5.019790946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.011117457 +0000 UTC m=+1378.748461685" watchObservedRunningTime="2026-11-03 22:45:05.019790946 +0000 UTC m=+1378.757135172" Nov 03 22:45:05.138378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.138332 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" podStartSLOduration=5.138319329 podStartE2EDuration="5.138319329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.132935103 +0000 UTC m=+1378.870279326" watchObservedRunningTime="2026-11-03 22:45:05.138319329 +0000 UTC m=+1378.875663553" Nov 03 22:45:05.204177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.204143 5051 generic.go:356] "Generic (PLEG): container finished" podID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerID="6e9e15ccd44d3ffe30a9484bc2083844281987f0fe7295a3bd1f7b84ad3f5a32" exitCode=60 Nov 03 22:45:05.204423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.204396 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerDied","Data":"6e9e15ccd44d3ffe30a9484bc2083844281987f0fe7295a3bd1f7b84ad3f5a32"} Nov 03 22:45:05.205026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.205006 5051 scope.go:122] "RemoveContainer" containerID="6e9e15ccd44d3ffe30a9484bc2083844281987f0fe7295a3bd1f7b84ad3f5a32" Nov 03 22:45:05.210471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.210446 5051 generic.go:356] "Generic (PLEG): container finished" podID="66e20b50-08ac-4535-b46b-7069f062c933" containerID="1d4e8606f24e2291482ed4d710f5f5765e6fea61907e7b89510d5bc09fa32411" exitCode=60 Nov 03 22:45:05.210741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.210724 5051 scope.go:122] "RemoveContainer" containerID="1d4e8606f24e2291482ed4d710f5f5765e6fea61907e7b89510d5bc09fa32411" Nov 03 22:45:05.210972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.210504 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerDied","Data":"1d4e8606f24e2291482ed4d710f5f5765e6fea61907e7b89510d5bc09fa32411"} Nov 03 22:45:05.215066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.215040 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj"] Nov 03 22:45:05.216658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.216635 5051 generic.go:356] "Generic (PLEG): container finished" podID="099d9638-1b14-422c-8c87-396d8142ac0a" containerID="5333cf7cadb578314bc49c611e69bbac77f94a2b4f8450700123ef322fa0fe22" exitCode=60 Nov 03 22:45:05.217043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.217027 5051 scope.go:122] "RemoveContainer" containerID="5333cf7cadb578314bc49c611e69bbac77f94a2b4f8450700123ef322fa0fe22" Nov 03 22:45:05.219697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.219674 5051 generic.go:356] "Generic (PLEG): container finished" podID="b319845c-053b-4795-b6b6-1de753410ea2" containerID="425f39a8eded6371c4170c50882cfcd8934a71bb20a8db9beb5feb8af4f41529" exitCode=60 Nov 03 22:45:05.220617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.220602 5051 scope.go:122] "RemoveContainer" containerID="425f39a8eded6371c4170c50882cfcd8934a71bb20a8db9beb5feb8af4f41529" Nov 03 22:45:05.221372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.221350 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerDied","Data":"5333cf7cadb578314bc49c611e69bbac77f94a2b4f8450700123ef322fa0fe22"} Nov 03 22:45:05.221489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.221418 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerDied","Data":"425f39a8eded6371c4170c50882cfcd8934a71bb20a8db9beb5feb8af4f41529"} Nov 03 22:45:05.221556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.221500 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.222564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.222543 5051 generic.go:356] "Generic (PLEG): container finished" podID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerID="e323fcf484748c1e8c8f5a120ec4d85d8a46fd30cb222c6b79392c2fc6ce1682" exitCode=60 Nov 03 22:45:05.222655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.222599 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerDied","Data":"e323fcf484748c1e8c8f5a120ec4d85d8a46fd30cb222c6b79392c2fc6ce1682"} Nov 03 22:45:05.222857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.222841 5051 scope.go:122] "RemoveContainer" containerID="e323fcf484748c1e8c8f5a120ec4d85d8a46fd30cb222c6b79392c2fc6ce1682" Nov 03 22:45:05.246141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.246116 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:45:05.254919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.254898 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.279353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.278190 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:45:05.282649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.282107 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:45:05.292853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.292828 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj"] Nov 03 22:45:05.312111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.308446 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fc129ccf\"" Nov 03 22:45:05.314908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.314886 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:45:05.317569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.317421 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.317950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.317924 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwf2\" (UniqueName: \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.318695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.318673 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.359014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.356374 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fc129ccf-dockercfg-99jjf\"" Nov 03 22:45:05.359014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.356443 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:05.425016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.424234 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj46p\" (UniqueName: \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.425016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.424305 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.425016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.424456 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwf2\" (UniqueName: \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.425016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.424510 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.425016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.424540 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.435876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.430594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.440359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.439755 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:45:05.484070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.481915 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.504216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.502713 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.565624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.565545 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.565955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.565919 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj46p\" (UniqueName: \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.566839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.566820 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.567130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.567115 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdffs\" (UniqueName: \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.569132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.569108 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.572288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.572221 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:45:05.678319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.671224 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwf2\" (UniqueName: \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\") pod \"maas-api-revoke-sub-5994de05-7nrnj\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.678319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.676074 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdffs\" (UniqueName: \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.678319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.676202 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.678319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.676734 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj46p\" (UniqueName: \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\") pod \"payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.678668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.678488 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.732771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.732743 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdffs\" (UniqueName: \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\") pod \"payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:05.903311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.903280 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:05.908182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.906837 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:05.909365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:05.909334 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:06.234787 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.234714 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerStarted","Data":"911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc"} Nov 03 22:45:06.236699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.236673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerStarted","Data":"252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035"} Nov 03 22:45:06.238644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.238619 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerStarted","Data":"b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307"} Nov 03 22:45:06.240424 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.240405 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerStarted","Data":"3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070"} Nov 03 22:45:06.242343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.242323 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerStarted","Data":"0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3"} Nov 03 22:45:06.316276 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:06.316247 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03571ac3_ce57_4961_95d6_8aa8259db43a.slice/crio-323a5d7b20f6daddeace756705ab10fb8e977ca7a6c70434732d9c76bb28ebba WatchSource:0}: Error finding container 323a5d7b20f6daddeace756705ab10fb8e977ca7a6c70434732d9c76bb28ebba: Status 404 returned error can't find the container with id 323a5d7b20f6daddeace756705ab10fb8e977ca7a6c70434732d9c76bb28ebba Nov 03 22:45:06.317316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.317293 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:45:06.383626 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:06.383607 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350b5aab_ff83_4af5_8d9c_aca817d3cec4.slice/crio-935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622 WatchSource:0}: Error finding container 935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622: Status 404 returned error can't find the container with id 935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622 Nov 03 22:45:06.387783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.387762 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj"] Nov 03 22:45:06.410435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.410388 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" podStartSLOduration=6.410373076 podStartE2EDuration="6.410373076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.410114222 +0000 UTC m=+1380.147458449" watchObservedRunningTime="2026-11-03 22:45:06.410373076 +0000 UTC m=+1380.147717301" Nov 03 22:45:06.472969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.472854 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:45:06.527782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.527730 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" podStartSLOduration=6.52771204 podStartE2EDuration="6.52771204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.526314653 +0000 UTC m=+1380.263658876" watchObservedRunningTime="2026-11-03 22:45:06.52771204 +0000 UTC m=+1380.265056266" Nov 03 22:45:06.870854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:06.870803 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" podStartSLOduration=6.870787462 podStartE2EDuration="6.870787462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.868511276 +0000 UTC m=+1380.605855501" watchObservedRunningTime="2026-11-03 22:45:06.870787462 +0000 UTC m=+1380.608131686" Nov 03 22:45:07.255199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.255115 5051 generic.go:356] "Generic (PLEG): container finished" podID="66e20b50-08ac-4535-b46b-7069f062c933" containerID="252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035" exitCode=60 Nov 03 22:45:07.255199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.255189 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerDied","Data":"252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035"} Nov 03 22:45:07.255728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.255290 5051 scope.go:122] "RemoveContainer" containerID="1d4e8606f24e2291482ed4d710f5f5765e6fea61907e7b89510d5bc09fa32411" Nov 03 22:45:07.255728 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.255523 5051 scope.go:122] "RemoveContainer" containerID="252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035" Nov 03 22:45:07.255947 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:07.255916 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx_odh-ai-gateway-infra(66e20b50-08ac-4535-b46b-7069f062c933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" podUID="66e20b50-08ac-4535-b46b-7069f062c933" Nov 03 22:45:07.257505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.257477 5051 generic.go:356] "Generic (PLEG): container finished" podID="099d9638-1b14-422c-8c87-396d8142ac0a" containerID="b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307" exitCode=60 Nov 03 22:45:07.257619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.257558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerDied","Data":"b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307"} Nov 03 22:45:07.257853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.257838 5051 scope.go:122] "RemoveContainer" containerID="b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307" Nov 03 22:45:07.258252 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:07.258214 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85_odh-ai-gateway-infra(099d9638-1b14-422c-8c87-396d8142ac0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" Nov 03 22:45:07.259861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.259837 5051 generic.go:356] "Generic (PLEG): container finished" podID="b319845c-053b-4795-b6b6-1de753410ea2" containerID="3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070" exitCode=60 Nov 03 22:45:07.259972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.259927 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerDied","Data":"3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070"} Nov 03 22:45:07.260259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.260202 5051 scope.go:122] "RemoveContainer" containerID="3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070" Nov 03 22:45:07.260746 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:07.260724 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf_odh-ai-gateway-infra(b319845c-053b-4795-b6b6-1de753410ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" podUID="b319845c-053b-4795-b6b6-1de753410ea2" Nov 03 22:45:07.262798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.262773 5051 generic.go:356] "Generic (PLEG): container finished" podID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerID="0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3" exitCode=60 Nov 03 22:45:07.262912 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.262816 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerDied","Data":"0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3"} Nov 03 22:45:07.263153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.263131 5051 scope.go:122] "RemoveContainer" containerID="0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3" Nov 03 22:45:07.263584 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:07.263559 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7_odh-ai-gateway-infra(28adff4f-6a5b-47a4-87ac-570f97a54953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" Nov 03 22:45:07.265233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.265204 5051 generic.go:356] "Generic (PLEG): container finished" podID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerID="911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc" exitCode=60 Nov 03 22:45:07.265343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.265279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerDied","Data":"911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc"} Nov 03 22:45:07.265532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.265517 5051 scope.go:122] "RemoveContainer" containerID="911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc" Nov 03 22:45:07.265814 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:07.265792 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc_odh-ai-gateway-infra(cad2bb11-a90c-4670-b3bf-9e2645b4ff42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" Nov 03 22:45:07.266936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.266911 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" event={"ID":"350b5aab-ff83-4af5-8d9c-aca817d3cec4","Type":"ContainerStarted","Data":"e153a47784d5e48b22672669d0e4868d4ee93fabb45c001a7ddb7c1ca5182a3a"} Nov 03 22:45:07.267114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.266953 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" event={"ID":"350b5aab-ff83-4af5-8d9c-aca817d3cec4","Type":"ContainerStarted","Data":"935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622"} Nov 03 22:45:07.268747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.268717 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" event={"ID":"386c549f-29ec-43eb-bdb5-81177026e39f","Type":"ContainerStarted","Data":"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b"} Nov 03 22:45:07.268747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.268749 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" event={"ID":"386c549f-29ec-43eb-bdb5-81177026e39f","Type":"ContainerStarted","Data":"5c22d930c7ed9664df825a2e0572a18f7495edd2d4afd2ca058f627486549558"} Nov 03 22:45:07.268890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.268786 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:07.270981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.270954 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" event={"ID":"03571ac3-ce57-4961-95d6-8aa8259db43a","Type":"ContainerStarted","Data":"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f"} Nov 03 22:45:07.270981 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.271023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" event={"ID":"03571ac3-ce57-4961-95d6-8aa8259db43a","Type":"ContainerStarted","Data":"323a5d7b20f6daddeace756705ab10fb8e977ca7a6c70434732d9c76bb28ebba"} Nov 03 22:45:07.271558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.271541 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:07.273817 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.273791 5051 scope.go:122] "RemoveContainer" containerID="5333cf7cadb578314bc49c611e69bbac77f94a2b4f8450700123ef322fa0fe22" Nov 03 22:45:07.294004 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.293976 5051 scope.go:122] "RemoveContainer" containerID="425f39a8eded6371c4170c50882cfcd8934a71bb20a8db9beb5feb8af4f41529" Nov 03 22:45:07.309122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.309068 5051 scope.go:122] "RemoveContainer" containerID="e323fcf484748c1e8c8f5a120ec4d85d8a46fd30cb222c6b79392c2fc6ce1682" Nov 03 22:45:07.325139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.324982 5051 scope.go:122] "RemoveContainer" containerID="6e9e15ccd44d3ffe30a9484bc2083844281987f0fe7295a3bd1f7b84ad3f5a32" Nov 03 22:45:07.609144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.609091 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" podStartSLOduration=2.6090743 podStartE2EDuration="2.6090743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.60561392 +0000 UTC m=+1381.342958144" watchObservedRunningTime="2026-11-03 22:45:07.6090743 +0000 UTC m=+1381.346418524" Nov 03 22:45:07.667565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.667533 5051 scope.go:122] "RemoveContainer" containerID="cac533a2e02fc28e27c78140f4214317ed72d6ad6ea023fc37ed5a141ba520a3" Nov 03 22:45:07.727158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.727137 5051 scope.go:122] "RemoveContainer" containerID="fa1b35cb5a99ca3aa1236f4d4dde4e510c320a5d860d04bbee8d9c5fc37ff74e" Nov 03 22:45:07.739739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.739713 5051 scope.go:122] "RemoveContainer" containerID="5ed192c0406007a1742ed0f837c6dc5f444822a83d1af54cd2ed66f1fd21f209" Nov 03 22:45:07.753247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.753230 5051 scope.go:122] "RemoveContainer" containerID="ad6d22dcaee7c0ebcc57312d52f74aba0f0a1a34754bf6e4c2a72fc49a1f388b" Nov 03 22:45:07.763771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.763752 5051 scope.go:122] "RemoveContainer" containerID="60ca547adca6ffa38110f5ec148769193392377a1741d5cc4426c5dbe670effd" Nov 03 22:45:07.777138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.777112 5051 scope.go:122] "RemoveContainer" containerID="adfaa95479b431adccf5002c1badf4e6595a84afbd9505d3231db4e4ff272e3c" Nov 03 22:45:07.788203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.788178 5051 scope.go:122] "RemoveContainer" containerID="4864b0f474dd4804617fec5018de53e1bc1c78b5a325ab6abcc1906a1c6c151e" Nov 03 22:45:07.802634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.802612 5051 scope.go:122] "RemoveContainer" containerID="fbf24e005fae124285e9405355b045b95ec1e74856c884a79c7114d63062b194" Nov 03 22:45:07.812303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.812285 5051 scope.go:122] "RemoveContainer" containerID="871b8ec096d8ae8e97799c5c3585be279aa39c1359df477aa115a0a49bd12132" Nov 03 22:45:07.821625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.821609 5051 scope.go:122] "RemoveContainer" containerID="1536dcc407cbeca0616f928be28a5bbaec606738efc9b8916a108946caa89766" Nov 03 22:45:07.966189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.966126 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" podStartSLOduration=2.966106772 podStartE2EDuration="2.966106772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.838250182 +0000 UTC m=+1381.575594406" watchObservedRunningTime="2026-11-03 22:45:07.966106772 +0000 UTC m=+1381.703451010" Nov 03 22:45:07.968856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:07.968806 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" podStartSLOduration=2.968793738 podStartE2EDuration="2.968793738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:07.951926672 +0000 UTC m=+1381.689270896" watchObservedRunningTime="2026-11-03 22:45:07.968793738 +0000 UTC m=+1381.706137963" Nov 03 22:45:08.302660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.302575 5051 scope.go:122] "RemoveContainer" containerID="3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070" Nov 03 22:45:08.303131 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:08.303018 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf_odh-ai-gateway-infra(b319845c-053b-4795-b6b6-1de753410ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" podUID="b319845c-053b-4795-b6b6-1de753410ea2" Nov 03 22:45:08.304508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.304480 5051 scope.go:122] "RemoveContainer" containerID="0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3" Nov 03 22:45:08.304882 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:08.304858 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7_odh-ai-gateway-infra(28adff4f-6a5b-47a4-87ac-570f97a54953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" Nov 03 22:45:08.306506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.306478 5051 scope.go:122] "RemoveContainer" containerID="911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc" Nov 03 22:45:08.307575 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:08.306756 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc_odh-ai-gateway-infra(cad2bb11-a90c-4670-b3bf-9e2645b4ff42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" Nov 03 22:45:08.308847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.308823 5051 generic.go:356] "Generic (PLEG): container finished" podID="350b5aab-ff83-4af5-8d9c-aca817d3cec4" containerID="e153a47784d5e48b22672669d0e4868d4ee93fabb45c001a7ddb7c1ca5182a3a" exitCode=0 Nov 03 22:45:08.308965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.308920 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" event={"ID":"350b5aab-ff83-4af5-8d9c-aca817d3cec4","Type":"ContainerDied","Data":"e153a47784d5e48b22672669d0e4868d4ee93fabb45c001a7ddb7c1ca5182a3a"} Nov 03 22:45:08.311637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.311613 5051 scope.go:122] "RemoveContainer" containerID="252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035" Nov 03 22:45:08.311936 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:08.311912 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx_odh-ai-gateway-infra(66e20b50-08ac-4535-b46b-7069f062c933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" podUID="66e20b50-08ac-4535-b46b-7069f062c933" Nov 03 22:45:08.314901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:08.314878 5051 scope.go:122] "RemoveContainer" containerID="b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307" Nov 03 22:45:08.315218 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:08.315200 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85_odh-ai-gateway-infra(099d9638-1b14-422c-8c87-396d8142ac0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" Nov 03 22:45:09.524515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.524492 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:09.651500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.651472 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\") pod \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " Nov 03 22:45:09.651665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.651641 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\") pod \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " Nov 03 22:45:09.651748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.651728 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\" (UniqueName: \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\") pod \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\" (UID: \"350b5aab-ff83-4af5-8d9c-aca817d3cec4\") " Nov 03 22:45:09.651879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.651855 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca" pod "350b5aab-ff83-4af5-8d9c-aca817d3cec4" (UID: "350b5aab-ff83-4af5-8d9c-aca817d3cec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:09.652436 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.652411 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:09.656446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.656376 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca" pod "350b5aab-ff83-4af5-8d9c-aca817d3cec4" (UID: "350b5aab-ff83-4af5-8d9c-aca817d3cec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:09.656446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.656388 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2" pod "350b5aab-ff83-4af5-8d9c-aca817d3cec4" (UID: "350b5aab-ff83-4af5-8d9c-aca817d3cec4"). InnerVolumeSpecName "kube-api-access-6fwf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:09.753755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.753727 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350b5aab-ff83-4af5-8d9c-aca817d3cec4-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:09.753755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:09.753754 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fwf2\" (UniqueName: \"kubernetes.io/projected/350b5aab-ff83-4af5-8d9c-aca817d3cec4-kube-api-access-6fwf2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:10.334092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:10.334062 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" Nov 03 22:45:10.334270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:10.334095 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj" event={"ID":"350b5aab-ff83-4af5-8d9c-aca817d3cec4","Type":"ContainerDied","Data":"935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622"} Nov 03 22:45:10.334270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:10.334143 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="935a331525dbeff63d245c160047dd6c218725e430d89e4711c6eaa345ae5622" Nov 03 22:45:15.953201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:15.953130 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll"] Nov 03 22:45:15.953848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:15.953833 5051 cpu_manager.go:414] "containerMap: removing container" podUID="350b5aab-ff83-4af5-8d9c-aca817d3cec4" containerName="revoke-keys" Nov 03 22:45:15.953886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:15.953850 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350b5aab-ff83-4af5-8d9c-aca817d3cec4" containerName="revoke-keys" Nov 03 22:45:15.954026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:15.954016 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="350b5aab-ff83-4af5-8d9c-aca817d3cec4" containerName="revoke-keys" Nov 03 22:45:15.958952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:15.958928 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.016085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.016056 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll"] Nov 03 22:45:16.040057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.039984 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.040181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.040115 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt8v\" (UniqueName: \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.040273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.040259 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.058216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.058185 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:16.142196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.142163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt8v\" (UniqueName: \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.142334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.142315 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.142520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.142503 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.142999 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.142968 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.144947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.144926 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.202926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.202901 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt8v\" (UniqueName: \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\") pod \"maas-api-revoke-sub-9d8f44bd-hcmll\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.277036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.276901 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:16.678445 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:16.678411 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26b6252_36e3_4fd7_8b89_6fd7fb902a10.slice/crio-c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e WatchSource:0}: Error finding container c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e: Status 404 returned error can't find the container with id c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e Nov 03 22:45:16.679321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:16.679297 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll"] Nov 03 22:45:17.414663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:17.414626 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" event={"ID":"a26b6252-36e3-4fd7-8b89-6fd7fb902a10","Type":"ContainerStarted","Data":"3ac816f04c6d7afd03fe9d00c4e48d941422aa0ad449c1004536ff5e6064d5d6"} Nov 03 22:45:17.415097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:17.414675 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" event={"ID":"a26b6252-36e3-4fd7-8b89-6fd7fb902a10","Type":"ContainerStarted","Data":"c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e"} Nov 03 22:45:17.537827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:17.537780 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" podStartSLOduration=2.5377674409999997 podStartE2EDuration="2.537767441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:17.534888125 +0000 UTC m=+1391.272232349" watchObservedRunningTime="2026-11-03 22:45:17.537767441 +0000 UTC m=+1391.275111664" Nov 03 22:45:18.325226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:18.325189 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:45:18.409066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:18.409037 5051 scope.go:122] "RemoveContainer" containerID="0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3" Nov 03 22:45:18.409284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:18.409266 5051 scope.go:122] "RemoveContainer" containerID="252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035" Nov 03 22:45:18.433844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:18.433813 5051 generic.go:356] "Generic (PLEG): container finished" podID="a26b6252-36e3-4fd7-8b89-6fd7fb902a10" containerID="3ac816f04c6d7afd03fe9d00c4e48d941422aa0ad449c1004536ff5e6064d5d6" exitCode=0 Nov 03 22:45:18.434215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:18.433896 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" event={"ID":"a26b6252-36e3-4fd7-8b89-6fd7fb902a10","Type":"ContainerDied","Data":"3ac816f04c6d7afd03fe9d00c4e48d941422aa0ad449c1004536ff5e6064d5d6"} Nov 03 22:45:19.327021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.326966 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:45:19.401405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.401376 5051 scope.go:122] "RemoveContainer" containerID="b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307" Nov 03 22:45:19.448130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.448095 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerStarted","Data":"74fe8349f578754a5084b96b98e7c6d3a5a153958309cf55a7f9c2249a4bc4ad"} Nov 03 22:45:19.451288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.451262 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerStarted","Data":"f5132154dc261404097c95b02a0258808b2a1080f17bd7ad29db3d7a616ebf18"} Nov 03 22:45:19.868960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.868935 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:19.909817 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.909788 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\") pod \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " Nov 03 22:45:19.909958 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.909926 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\" (UniqueName: \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\") pod \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " Nov 03 22:45:19.910024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.909976 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\") pod \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\" (UID: \"a26b6252-36e3-4fd7-8b89-6fd7fb902a10\") " Nov 03 22:45:19.910349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.910324 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca" pod "a26b6252-36e3-4fd7-8b89-6fd7fb902a10" (UID: "a26b6252-36e3-4fd7-8b89-6fd7fb902a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:19.910643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.910627 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:19.912700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.912674 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca" pod "a26b6252-36e3-4fd7-8b89-6fd7fb902a10" (UID: "a26b6252-36e3-4fd7-8b89-6fd7fb902a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:19.912700 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:19.912686 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v" pod "a26b6252-36e3-4fd7-8b89-6fd7fb902a10" (UID: "a26b6252-36e3-4fd7-8b89-6fd7fb902a10"). InnerVolumeSpecName "kube-api-access-zjt8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.012128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.012093 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjt8v\" (UniqueName: \"kubernetes.io/projected/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-kube-api-access-zjt8v\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.012128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.012130 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26b6252-36e3-4fd7-8b89-6fd7fb902a10-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.208856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.208780 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:45:20.209602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.209584 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a26b6252-36e3-4fd7-8b89-6fd7fb902a10" containerName="revoke-keys" Nov 03 22:45:20.209670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.209603 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26b6252-36e3-4fd7-8b89-6fd7fb902a10" containerName="revoke-keys" Nov 03 22:45:20.209735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.209726 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26b6252-36e3-4fd7-8b89-6fd7fb902a10" containerName="revoke-keys" Nov 03 22:45:20.213543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.213527 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.316679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.316642 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.316838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.316775 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.316838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.316817 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.316979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.316842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.316979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.316940 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.317120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.317018 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdl4\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.317176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.317153 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.317234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.317178 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.317234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.317205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.341399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.341370 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:45:20.401784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.401762 5051 scope.go:122] "RemoveContainer" containerID="3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070" Nov 03 22:45:20.401947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.401930 5051 scope.go:122] "RemoveContainer" containerID="911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc" Nov 03 22:45:20.418427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418406 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418436 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418468 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418606 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418748 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418816 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418822 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418848 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418878 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.418937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.418922 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdl4\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.419350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.419329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.419479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.419456 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.419543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.419476 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.419736 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.419719 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.421815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.421793 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.422185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.422165 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.463741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.463692 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" Nov 03 22:45:20.463741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.463727 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll" event={"ID":"a26b6252-36e3-4fd7-8b89-6fd7fb902a10","Type":"ContainerDied","Data":"c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e"} Nov 03 22:45:20.464218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.463768 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37f6a1a50369f4e1c592fc18a62c3c8d82ffaddcba3aab7f610fd72ec74032e" Nov 03 22:45:20.466618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.466451 5051 generic.go:356] "Generic (PLEG): container finished" podID="66e20b50-08ac-4535-b46b-7069f062c933" containerID="74fe8349f578754a5084b96b98e7c6d3a5a153958309cf55a7f9c2249a4bc4ad" exitCode=60 Nov 03 22:45:20.466618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.466549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerDied","Data":"74fe8349f578754a5084b96b98e7c6d3a5a153958309cf55a7f9c2249a4bc4ad"} Nov 03 22:45:20.466618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.466591 5051 scope.go:122] "RemoveContainer" containerID="252d51c8c2b5c09eeb65bedf1623144e321aa2cb414434e411f756756b920035" Nov 03 22:45:20.467082 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.467057 5051 scope.go:122] "RemoveContainer" containerID="74fe8349f578754a5084b96b98e7c6d3a5a153958309cf55a7f9c2249a4bc4ad" Nov 03 22:45:20.467391 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:20.467368 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx_odh-ai-gateway-infra(66e20b50-08ac-4535-b46b-7069f062c933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" podUID="66e20b50-08ac-4535-b46b-7069f062c933" Nov 03 22:45:20.469044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.468974 5051 generic.go:356] "Generic (PLEG): container finished" podID="099d9638-1b14-422c-8c87-396d8142ac0a" containerID="5d10a23966ef3d6376136c1819be6e9df5e41c6cc14405c5ba7dd721b84b51cc" exitCode=60 Nov 03 22:45:20.469116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.469042 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerDied","Data":"5d10a23966ef3d6376136c1819be6e9df5e41c6cc14405c5ba7dd721b84b51cc"} Nov 03 22:45:20.469395 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.469373 5051 scope.go:122] "RemoveContainer" containerID="5d10a23966ef3d6376136c1819be6e9df5e41c6cc14405c5ba7dd721b84b51cc" Nov 03 22:45:20.469673 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:20.469650 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85_odh-ai-gateway-infra(099d9638-1b14-422c-8c87-396d8142ac0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" Nov 03 22:45:20.471219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.471199 5051 generic.go:356] "Generic (PLEG): container finished" podID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerID="f5132154dc261404097c95b02a0258808b2a1080f17bd7ad29db3d7a616ebf18" exitCode=60 Nov 03 22:45:20.471296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.471227 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerDied","Data":"f5132154dc261404097c95b02a0258808b2a1080f17bd7ad29db3d7a616ebf18"} Nov 03 22:45:20.471591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.471577 5051 scope.go:122] "RemoveContainer" containerID="f5132154dc261404097c95b02a0258808b2a1080f17bd7ad29db3d7a616ebf18" Nov 03 22:45:20.471869 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:20.471850 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7_odh-ai-gateway-infra(28adff4f-6a5b-47a4-87ac-570f97a54953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" Nov 03 22:45:20.483479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.483459 5051 scope.go:122] "RemoveContainer" containerID="b13a222326333ed130cee8f169c4dbfa7ba728c5d9b62b9d92514cb7a7051307" Nov 03 22:45:20.500295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.500276 5051 scope.go:122] "RemoveContainer" containerID="0b788159de85bc6615e2b59e8a2c335e3b07dcf9694f05687b00b2ed4be512e3" Nov 03 22:45:20.557962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.557941 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.558267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.558247 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdl4\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.597930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.597897 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:20.692142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.692116 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:20.777215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.777190 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:45:20.781835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.781808 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825036 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825123 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825177 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825207 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825326 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825382 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlgq\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825473 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.825613 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.825564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.828225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.828199 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:20.854032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.847641 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927305 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927488 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927545 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927602 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlgq\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927632 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927724 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927844 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928055 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.927956 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.928266 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.928503 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.928736 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.928935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.928771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.931225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.931167 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.931852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.931829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.989784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.989756 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:20.990110 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:20.990085 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlgq\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:21.096855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.096768 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:21.297300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.297267 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:45:21.301769 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:21.301737 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de53ca3_628d_492a_bc0f_69387afd86d2.slice/crio-d8b5fff26a1944d2b6ebbdb95e5441d2b0456bfc91ba2bb85fb9d80a9439d22f WatchSource:0}: Error finding container d8b5fff26a1944d2b6ebbdb95e5441d2b0456bfc91ba2bb85fb9d80a9439d22f: Status 404 returned error can't find the container with id d8b5fff26a1944d2b6ebbdb95e5441d2b0456bfc91ba2bb85fb9d80a9439d22f Nov 03 22:45:21.304632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.304595 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.304719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.304666 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.304719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.304695 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.487782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.487752 5051 generic.go:356] "Generic (PLEG): container finished" podID="b319845c-053b-4795-b6b6-1de753410ea2" containerID="2e78ca05a25efc6e450e98cd5392be936b2470c9ba042bc6e5482380b6c931f3" exitCode=60 Nov 03 22:45:21.488248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.487833 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerDied","Data":"2e78ca05a25efc6e450e98cd5392be936b2470c9ba042bc6e5482380b6c931f3"} Nov 03 22:45:21.488248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.487911 5051 scope.go:122] "RemoveContainer" containerID="3e96ea8952ec96b6fe9087956053a7ab406cadc3ad455b4e92deb7ee5373f070" Nov 03 22:45:21.488464 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.488445 5051 scope.go:122] "RemoveContainer" containerID="2e78ca05a25efc6e450e98cd5392be936b2470c9ba042bc6e5482380b6c931f3" Nov 03 22:45:21.488910 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:21.488888 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf_odh-ai-gateway-infra(b319845c-053b-4795-b6b6-1de753410ea2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" podUID="b319845c-053b-4795-b6b6-1de753410ea2" Nov 03 22:45:21.494881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.493919 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" event={"ID":"8de53ca3-628d-492a-bc0f-69387afd86d2","Type":"ContainerStarted","Data":"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4"} Nov 03 22:45:21.494881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.493969 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" event={"ID":"8de53ca3-628d-492a-bc0f-69387afd86d2","Type":"ContainerStarted","Data":"d8b5fff26a1944d2b6ebbdb95e5441d2b0456bfc91ba2bb85fb9d80a9439d22f"} Nov 03 22:45:21.495637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.495498 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:45:21.498149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.498125 5051 generic.go:356] "Generic (PLEG): container finished" podID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerID="e57628183fd39da9d0054dd9b9fc27772702fc8e87e0e8667eaf66bf20f0158b" exitCode=60 Nov 03 22:45:21.498253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.498229 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerDied","Data":"e57628183fd39da9d0054dd9b9fc27772702fc8e87e0e8667eaf66bf20f0158b"} Nov 03 22:45:21.498670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.498632 5051 scope.go:122] "RemoveContainer" containerID="e57628183fd39da9d0054dd9b9fc27772702fc8e87e0e8667eaf66bf20f0158b" Nov 03 22:45:21.499204 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:21.499176 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc_odh-ai-gateway-infra(cad2bb11-a90c-4670-b3bf-9e2645b4ff42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" Nov 03 22:45:21.551654 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:21.551614 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8de30bd_5149_4e72_a5fd_1cdb838bbe2c.slice/crio-a6db5d342e9f73c704308a44e4adc9037be1adc6d8f55ccf14e4b87ca40589d9 WatchSource:0}: Error finding container a6db5d342e9f73c704308a44e4adc9037be1adc6d8f55ccf14e4b87ca40589d9: Status 404 returned error can't find the container with id a6db5d342e9f73c704308a44e4adc9037be1adc6d8f55ccf14e4b87ca40589d9 Nov 03 22:45:21.552061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.551965 5051 scope.go:122] "RemoveContainer" containerID="911b6d7bddcdd5769c79ba8cf98af28ac10d4bcc6915bafd4ef0b9689f7262bc" Nov 03 22:45:21.559343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.559136 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.559343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.559227 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.559343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.559273 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.667734 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:21.667697 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:45:21.730904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.730848 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:21.791955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.791875 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:21.829080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.829044 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:21.835274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.835250 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:21.844502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.844477 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\" (UniqueName: \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\") pod \"66e20b50-08ac-4535-b46b-7069f062c933\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " Nov 03 22:45:21.845557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.844553 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\") pod \"66e20b50-08ac-4535-b46b-7069f062c933\" (UID: \"66e20b50-08ac-4535-b46b-7069f062c933\") " Nov 03 22:45:21.845557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.845223 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca" pod "66e20b50-08ac-4535-b46b-7069f062c933" (UID: "66e20b50-08ac-4535-b46b-7069f062c933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.845557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.845392 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e20b50-08ac-4535-b46b-7069f062c933-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.851101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.851076 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp" pod "66e20b50-08ac-4535-b46b-7069f062c933" (UID: "66e20b50-08ac-4535-b46b-7069f062c933"). InnerVolumeSpecName "kube-api-access-xv4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.946627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.946520 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\" (UniqueName: \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\") pod \"28adff4f-6a5b-47a4-87ac-570f97a54953\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " Nov 03 22:45:21.947639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.946746 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\") pod \"28adff4f-6a5b-47a4-87ac-570f97a54953\" (UID: \"28adff4f-6a5b-47a4-87ac-570f97a54953\") " Nov 03 22:45:21.947639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.947489 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv4fp\" (UniqueName: \"kubernetes.io/projected/66e20b50-08ac-4535-b46b-7069f062c933-kube-api-access-xv4fp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.951169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.951122 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca" pod "28adff4f-6a5b-47a4-87ac-570f97a54953" (UID: "28adff4f-6a5b-47a4-87ac-570f97a54953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.955161 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.955133 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh" pod "28adff4f-6a5b-47a4-87ac-570f97a54953" (UID: "28adff4f-6a5b-47a4-87ac-570f97a54953"). InnerVolumeSpecName "kube-api-access-d2nxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.970885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.970666 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podStartSLOduration=1.970645496 podStartE2EDuration="1.970645496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:21.969707101 +0000 UTC m=+1395.707051329" watchObservedRunningTime="2026-11-03 22:45:21.970645496 +0000 UTC m=+1395.707989722" Nov 03 22:45:21.998144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:21.998112 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:22.048771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.048732 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\" (UniqueName: \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\") pod \"099d9638-1b14-422c-8c87-396d8142ac0a\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " Nov 03 22:45:22.048938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.048862 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\") pod \"099d9638-1b14-422c-8c87-396d8142ac0a\" (UID: \"099d9638-1b14-422c-8c87-396d8142ac0a\") " Nov 03 22:45:22.049233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.049213 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca" pod "099d9638-1b14-422c-8c87-396d8142ac0a" (UID: "099d9638-1b14-422c-8c87-396d8142ac0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.049431 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.049414 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/099d9638-1b14-422c-8c87-396d8142ac0a-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.049562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.049432 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nxh\" (UniqueName: \"kubernetes.io/projected/28adff4f-6a5b-47a4-87ac-570f97a54953-kube-api-access-d2nxh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.049562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.049442 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28adff4f-6a5b-47a4-87ac-570f97a54953-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.051279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.051253 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl" pod "099d9638-1b14-422c-8c87-396d8142ac0a" (UID: "099d9638-1b14-422c-8c87-396d8142ac0a"). InnerVolumeSpecName "kube-api-access-dglcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.151590 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.151553 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglcl\" (UniqueName: \"kubernetes.io/projected/099d9638-1b14-422c-8c87-396d8142ac0a-kube-api-access-dglcl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:22.542310 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.542271 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" event={"ID":"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c","Type":"ContainerStarted","Data":"0f2f96a4a9023bf9562b8841099178e341f20c133e93fdca8d84fe3352e10a9c"} Nov 03 22:45:22.542914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.542890 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" event={"ID":"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c","Type":"ContainerStarted","Data":"a6db5d342e9f73c704308a44e4adc9037be1adc6d8f55ccf14e4b87ca40589d9"} Nov 03 22:45:22.544748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.544728 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" event={"ID":"66e20b50-08ac-4535-b46b-7069f062c933","Type":"ContainerDied","Data":"c43cf6092d0ec033333f9964bc52d8b22c4b9aad59d9ae9edfb502e2cc1b4552"} Nov 03 22:45:22.544954 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.544934 5051 scope.go:122] "RemoveContainer" containerID="74fe8349f578754a5084b96b98e7c6d3a5a153958309cf55a7f9c2249a4bc4ad" Nov 03 22:45:22.545110 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.544843 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx" Nov 03 22:45:22.549951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.549930 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" event={"ID":"099d9638-1b14-422c-8c87-396d8142ac0a","Type":"ContainerDied","Data":"61619e2ee2318520ce226e6a6ba4994083f2fe62b2dcf952d931ea4910c758a5"} Nov 03 22:45:22.551328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.550469 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85" Nov 03 22:45:22.643947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.643919 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" Nov 03 22:45:22.644183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.644153 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7" event={"ID":"28adff4f-6a5b-47a4-87ac-570f97a54953","Type":"ContainerDied","Data":"e3e908dcfa8eb345e238d29f740fde7d02e2568c9530b5168ebc4860e61863d4"} Nov 03 22:45:22.652323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.652304 5051 scope.go:122] "RemoveContainer" containerID="5d10a23966ef3d6376136c1819be6e9df5e41c6cc14405c5ba7dd721b84b51cc" Nov 03 22:45:22.743516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.743430 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podStartSLOduration=2.743415288 podStartE2EDuration="2.743415288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:22.73951285 +0000 UTC m=+1396.476857077" watchObservedRunningTime="2026-11-03 22:45:22.743415288 +0000 UTC m=+1396.480759507" Nov 03 22:45:22.753782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.753655 5051 scope.go:122] "RemoveContainer" containerID="f5132154dc261404097c95b02a0258808b2a1080f17bd7ad29db3d7a616ebf18" Nov 03 22:45:22.754939 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.754696 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:22.834884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.834846 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.835125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.834898 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:22.942015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.940226 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:22.950013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:22.949000 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895765-lf5vx"] Nov 03 22:45:23.041108 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.041038 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:23.048793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.048765 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b64e2e-29895765-v5dx7"] Nov 03 22:45:23.136317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.135545 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:23.139627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.137881 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.139627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.137943 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:23.147250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.147228 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:23.229256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.229226 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:23.238504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.238464 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895765-cxf85"] Nov 03 22:45:23.454089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.454037 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:23.625605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.625563 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\" (UniqueName: \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\") pod \"b319845c-053b-4795-b6b6-1de753410ea2\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " Nov 03 22:45:23.626254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.626234 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\") pod \"b319845c-053b-4795-b6b6-1de753410ea2\" (UID: \"b319845c-053b-4795-b6b6-1de753410ea2\") " Nov 03 22:45:23.626721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.626616 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca" pod "b319845c-053b-4795-b6b6-1de753410ea2" (UID: "b319845c-053b-4795-b6b6-1de753410ea2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.627516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.627499 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319845c-053b-4795-b6b6-1de753410ea2-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.635328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.635303 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm" pod "b319845c-053b-4795-b6b6-1de753410ea2" (UID: "b319845c-053b-4795-b6b6-1de753410ea2"). InnerVolumeSpecName "kube-api-access-p97gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.729151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.729108 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97gm\" (UniqueName: \"kubernetes.io/projected/b319845c-053b-4795-b6b6-1de753410ea2-kube-api-access-p97gm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.729408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.729387 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" Nov 03 22:45:23.729514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.729423 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf" event={"ID":"b319845c-053b-4795-b6b6-1de753410ea2","Type":"ContainerDied","Data":"7c400b6c2ba4ac793e2187639fd6cdf849b1207f63aaf2c4bb39e5fdab994cf9"} Nov 03 22:45:23.729514 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.729496 5051 scope.go:122] "RemoveContainer" containerID="2e78ca05a25efc6e450e98cd5392be936b2470c9ba042bc6e5482380b6c931f3" Nov 03 22:45:23.733352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.733258 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:23.735498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.735472 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" event={"ID":"cad2bb11-a90c-4670-b3bf-9e2645b4ff42","Type":"ContainerDied","Data":"cd3f0cb93bdbafd89b941f0bc7731aa716072f71e7956a55f6e11e0489294b38"} Nov 03 22:45:23.745895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.745873 5051 scope.go:122] "RemoveContainer" containerID="e57628183fd39da9d0054dd9b9fc27772702fc8e87e0e8667eaf66bf20f0158b" Nov 03 22:45:23.828836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.828803 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.829033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.828849 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:23.829627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.829605 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\") pod \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " Nov 03 22:45:23.829717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.829703 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\" (UniqueName: \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\") pod \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\" (UID: \"cad2bb11-a90c-4670-b3bf-9e2645b4ff42\") " Nov 03 22:45:23.829979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.829955 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca" pod "cad2bb11-a90c-4670-b3bf-9e2645b4ff42" (UID: "cad2bb11-a90c-4670-b3bf-9e2645b4ff42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.831079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.831023 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:23.833506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.833479 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55" pod "cad2bb11-a90c-4670-b3bf-9e2645b4ff42" (UID: "cad2bb11-a90c-4670-b3bf-9e2645b4ff42"). InnerVolumeSpecName "kube-api-access-vbl55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.933031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:23.932933 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbl55\" (UniqueName: \"kubernetes.io/projected/cad2bb11-a90c-4670-b3bf-9e2645b4ff42-kube-api-access-vbl55\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:24.106455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.106421 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.122454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.106677 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:24.125427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.125398 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:24.130836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.130775 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-164c36-29895765-fn9pf"] Nov 03 22:45:24.428476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.427091 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" path="/var/lib/kubelet/pods/099d9638-1b14-422c-8c87-396d8142ac0a/volumes" Nov 03 22:45:24.428476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.427568 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" path="/var/lib/kubelet/pods/28adff4f-6a5b-47a4-87ac-570f97a54953/volumes" Nov 03 22:45:24.428476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.428175 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e20b50-08ac-4535-b46b-7069f062c933" path="/var/lib/kubelet/pods/66e20b50-08ac-4535-b46b-7069f062c933/volumes" Nov 03 22:45:24.428952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.428933 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b319845c-053b-4795-b6b6-1de753410ea2" path="/var/lib/kubelet/pods/b319845c-053b-4795-b6b6-1de753410ea2/volumes" Nov 03 22:45:24.747828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.747747 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc" Nov 03 22:45:24.829541 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.829504 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.829719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.829566 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:24.932368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.932330 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:24.938827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:24.938780 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-32449d-29895765-fklmc"] Nov 03 22:45:25.110117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:25.097400 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.110117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:25.097476 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:25.828632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:25.828596 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.829050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:25.828651 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:26.116016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:26.111320 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.116016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:26.111385 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:26.451465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:26.451385 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" path="/var/lib/kubelet/pods/cad2bb11-a90c-4670-b3bf-9e2645b4ff42/volumes" Nov 03 22:45:26.829305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:26.829218 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.829870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:26.829836 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:27.112024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:27.111389 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.112024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:27.111450 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:27.829313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:27.829268 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.829860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:27.829325 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 22:45:28.097553 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:28.097448 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.097553 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:28.097526 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:45:29.102719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.101641 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:29.103702 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.103672 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:29.105247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.105212 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:45:29.410186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.410091 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:29.411222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.411193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:29.412905 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:29.412880 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:45:32.572806 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.572769 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr"] Nov 03 22:45:32.573540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573523 5051 cpu_manager.go:414] "containerMap: removing container" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.573540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573540 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573557 5051 cpu_manager.go:414] "containerMap: removing container" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573565 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573582 5051 cpu_manager.go:414] "containerMap: removing container" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573587 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573595 5051 cpu_manager.go:414] "containerMap: removing container" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573602 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573622 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573629 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573647 5051 cpu_manager.go:414] "containerMap: removing container" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573654 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573663 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573669 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573675 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573679 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573687 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.573686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573691 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573710 5051 cpu_manager.go:414] "containerMap: removing container" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573715 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573858 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573869 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573875 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573881 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573887 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573892 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573897 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573903 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573910 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573915 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573925 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:32.574546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.573930 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:32.579548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.579528 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.625940 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.625909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.626256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.626067 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrnq\" (UniqueName: \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.626348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.626328 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.634394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.634365 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr"] Nov 03 22:45:32.676191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.676152 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:32.678579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.678489 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:45:32.727841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.727801 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.728073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.727882 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.728073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.727946 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrnq\" (UniqueName: \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.728617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.728600 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.730501 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.730481 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.788898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.788860 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrnq\" (UniqueName: \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\") pod \"maas-api-revoke-sub-588c64a2-wg7kr\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:32.897316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:32.897087 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:33.292032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:33.291981 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr"] Nov 03 22:45:33.950658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:33.950566 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" event={"ID":"83a877ed-883c-4054-a214-097afb14700d","Type":"ContainerStarted","Data":"219035f316d0a413d40d7cd8f37fa448ad0f56091160743f079e4942b2d3c0bf"} Nov 03 22:45:33.950658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:33.950627 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" event={"ID":"83a877ed-883c-4054-a214-097afb14700d","Type":"ContainerStarted","Data":"0c50e2b305dea120e58eb8fe3339f8c37b8300503fa4616728cf286231700118"} Nov 03 22:45:34.072225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:34.072167 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" podStartSLOduration=2.072150595 podStartE2EDuration="2.072150595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:34.066260248 +0000 UTC m=+1407.803604476" watchObservedRunningTime="2026-11-03 22:45:34.072150595 +0000 UTC m=+1407.809494820" Nov 03 22:45:34.961414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:34.961378 5051 generic.go:356] "Generic (PLEG): container finished" podID="83a877ed-883c-4054-a214-097afb14700d" containerID="219035f316d0a413d40d7cd8f37fa448ad0f56091160743f079e4942b2d3c0bf" exitCode=0 Nov 03 22:45:34.961938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:34.961479 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" event={"ID":"83a877ed-883c-4054-a214-097afb14700d","Type":"ContainerDied","Data":"219035f316d0a413d40d7cd8f37fa448ad0f56091160743f079e4942b2d3c0bf"} Nov 03 22:45:35.900016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.899885 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905670 5051 cpu_manager.go:414] "containerMap: removing container" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905698 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905734 5051 cpu_manager.go:414] "containerMap: removing container" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905743 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905756 5051 cpu_manager.go:414] "containerMap: removing container" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905763 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905803 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905811 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad2bb11-a90c-4670-b3bf-9e2645b4ff42" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905828 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.905835 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b319845c-053b-4795-b6b6-1de753410ea2" containerName="cleanup" Nov 03 22:45:35.906021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.906036 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e20b50-08ac-4535-b46b-7069f062c933" containerName="cleanup" Nov 03 22:45:35.906747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.906068 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="28adff4f-6a5b-47a4-87ac-570f97a54953" containerName="cleanup" Nov 03 22:45:35.906747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.906081 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d9638-1b14-422c-8c87-396d8142ac0a" containerName="cleanup" Nov 03 22:45:35.931016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:35.924276 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.034472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.034038 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:45:36.055645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.055552 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.055645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.055601 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.055789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.055646 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.055789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.055726 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.055909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.055817 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjrz9\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.057545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.057505 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.057801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.057782 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.057970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.057950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.067575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.061158 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.162790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.162695 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.162790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.162759 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.162849 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.162912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.163037 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.163077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.163118 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.163203 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.163328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.163298 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjrz9\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.164159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.164130 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.164430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.164408 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.164678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.164657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.165339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.165317 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.166084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.166060 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.195356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.189789 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.195356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.195208 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.303421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.279823 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjrz9\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.303421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.297487 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.610031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.594212 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:36.676776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.676689 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:36.812019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.800579 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\") pod \"83a877ed-883c-4054-a214-097afb14700d\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " Nov 03 22:45:36.898198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.893846 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\" (UniqueName: \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\") pod \"83a877ed-883c-4054-a214-097afb14700d\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " Nov 03 22:45:36.898198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.894086 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\") pod \"83a877ed-883c-4054-a214-097afb14700d\" (UID: \"83a877ed-883c-4054-a214-097afb14700d\") " Nov 03 22:45:36.899301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.898717 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca" pod "83a877ed-883c-4054-a214-097afb14700d" (UID: "83a877ed-883c-4054-a214-097afb14700d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:36.910012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.907231 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83a877ed-883c-4054-a214-097afb14700d-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:36.983016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:36.980520 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca" pod "83a877ed-883c-4054-a214-097afb14700d" (UID: "83a877ed-883c-4054-a214-097afb14700d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:37.136016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.102750 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83a877ed-883c-4054-a214-097afb14700d-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:37.256019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.246267 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq" pod "83a877ed-883c-4054-a214-097afb14700d" (UID: "83a877ed-883c-4054-a214-097afb14700d"). InnerVolumeSpecName "kube-api-access-7lrnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:37.357015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.353982 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:37.399529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.392353 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrnq\" (UniqueName: \"kubernetes.io/projected/83a877ed-883c-4054-a214-097afb14700d-kube-api-access-7lrnq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:37.546018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.515200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:37.568016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.547062 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:37.660469 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.660371 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:37.913684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.909523 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" event={"ID":"83a877ed-883c-4054-a214-097afb14700d","Type":"ContainerDied","Data":"0c50e2b305dea120e58eb8fe3339f8c37b8300503fa4616728cf286231700118"} Nov 03 22:45:37.913684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.909615 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c50e2b305dea120e58eb8fe3339f8c37b8300503fa4616728cf286231700118" Nov 03 22:45:37.913684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.909929 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr" Nov 03 22:45:37.953633 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:37.953586 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.010180 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.010124 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 22:45:38.130732 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:45:38.080105 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eeffa_f6fd_47e7_a126_d55d80948ace.slice/crio-534ce9a44ea4b18db18792737a4eeb70ac180d7ff770a362402f048359237acd WatchSource:0}: Error finding container 534ce9a44ea4b18db18792737a4eeb70ac180d7ff770a362402f048359237acd: Status 404 returned error can't find the container with id 534ce9a44ea4b18db18792737a4eeb70ac180d7ff770a362402f048359237acd Nov 03 22:45:38.130732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.112680 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.230016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.206207 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:38.298023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.276118 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.330032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.329201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:45:38.389038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.378135 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.389038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.378421 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:45:38.389038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.379296 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:38.389038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.379361 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:38.389038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.380267 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:45:38.396036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.395210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:45:38.496828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.479548 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.576594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.576178 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-2wvgk" podUID="feb10c14-99bd-468d-932f-a4cb23eec994" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 22:45:38.611048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.578372 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.697169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.697101 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.698659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.698623 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Nov 03 22:45:38.749657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.727436 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.749657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.745133 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.750519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:38.750484 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.061002 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.060837 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.118367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.061643 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:39.118669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.118631 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.118801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.061715 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.118966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.061684 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.141551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.130182 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:39.141551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.130266 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.200145 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.200098 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded" Nov 03 22:45:39.272464 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.271125 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:45:39.298184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.298147 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:45:39.298481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.298467 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:45:39.380014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.378120 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:45:39.591805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.591757 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Nov 03 22:45:39.592447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.592416 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 22:45:39.595264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.595242 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:45:39.595445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.595431 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:45:39.689475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.687302 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 22:45:39.776406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.776191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.780700 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.786921 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.787250 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" podUID="a2e1a9bd-ec01-49a9-92ef-6688291fb567" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.787441 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.787604 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.787939 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:45:39.793054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.788014 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:45:39.826669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.826630 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" event={"ID":"0e4eeffa-f6fd-47e7-a126-d55d80948ace","Type":"ContainerStarted","Data":"534ce9a44ea4b18db18792737a4eeb70ac180d7ff770a362402f048359237acd"} Nov 03 22:45:39.956283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.956194 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 22:45:39.976559 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.966874 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:45:39.982054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:39.982028 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 22:45:40.055396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.054848 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 22:45:40.069560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.069529 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 22:45:40.193016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.192757 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 22:45:40.271539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.261553 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 22:45:40.288011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.286278 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 22:45:40.299132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.298449 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv" Nov 03 22:45:40.843559 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:40.843521 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" event={"ID":"0e4eeffa-f6fd-47e7-a126-d55d80948ace","Type":"ContainerStarted","Data":"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c"} Nov 03 22:45:41.614549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:41.614513 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:41.619104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:41.618786 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:41.619104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:41.618844 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:42.088503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:42.088394 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podStartSLOduration=7.08837695 podStartE2EDuration="7.08837695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:42.084452225 +0000 UTC m=+1415.821796452" watchObservedRunningTime="2026-11-03 22:45:42.08837695 +0000 UTC m=+1415.825721177" Nov 03 22:45:42.612542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:42.612509 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:42.612735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:42.612592 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:43.401969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.401844 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b"] Nov 03 22:45:43.403225 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.403144 5051 cpu_manager.go:414] "containerMap: removing container" podUID="83a877ed-883c-4054-a214-097afb14700d" containerName="revoke-keys" Nov 03 22:45:43.403412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.403393 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a877ed-883c-4054-a214-097afb14700d" containerName="revoke-keys" Nov 03 22:45:43.403711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.403691 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a877ed-883c-4054-a214-097afb14700d" containerName="revoke-keys" Nov 03 22:45:43.410980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.410959 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.442077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.442044 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b"] Nov 03 22:45:43.494715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.494684 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:43.495444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.495422 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:45:43.532063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.531965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.532223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.532138 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkffv\" (UniqueName: \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.532398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.532382 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.612585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.612555 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:43.612764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.612608 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:43.633935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.633898 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.634136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.633969 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkffv\" (UniqueName: \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.634136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.634124 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.634660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.634635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.636876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.636855 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.695169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.695095 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkffv\" (UniqueName: \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\") pod \"maas-api-revoke-sub-3b8743b6-wdq5b\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:43.734556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:43.734263 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:44.529057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:44.527403 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b"] Nov 03 22:45:44.612644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:44.612608 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:44.612822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:44.612661 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:45.020071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:45.020024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" event={"ID":"48d5c789-7603-4e49-b0f4-1d0835b6dc3c","Type":"ContainerStarted","Data":"3ea2b96071a4a8ac8cabd8667764826f7d000c09e51b562474a42f220f102f61"} Nov 03 22:45:45.020202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:45.020090 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" event={"ID":"48d5c789-7603-4e49-b0f4-1d0835b6dc3c","Type":"ContainerStarted","Data":"37794bf3ce5ada922874d1ca7772b057a016ae6e74d62735e5d66860c7ef5e0c"} Nov 03 22:45:45.141977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:45.141926 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" podStartSLOduration=2.141911597 podStartE2EDuration="2.141911597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:45.139160898 +0000 UTC m=+1418.876505124" watchObservedRunningTime="2026-11-03 22:45:45.141911597 +0000 UTC m=+1418.879255822" Nov 03 22:45:45.612574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:45.612536 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:45.612973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:45.612602 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:46.031774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:46.031683 5051 generic.go:356] "Generic (PLEG): container finished" podID="48d5c789-7603-4e49-b0f4-1d0835b6dc3c" containerID="3ea2b96071a4a8ac8cabd8667764826f7d000c09e51b562474a42f220f102f61" exitCode=0 Nov 03 22:45:46.031774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:46.031747 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" event={"ID":"48d5c789-7603-4e49-b0f4-1d0835b6dc3c","Type":"ContainerDied","Data":"3ea2b96071a4a8ac8cabd8667764826f7d000c09e51b562474a42f220f102f61"} Nov 03 22:45:46.613137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:46.613096 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 22:45:46.613550 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:46.613164 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 22:45:47.241095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.241069 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:47.329565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.329485 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\" (UniqueName: \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\") pod \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " Nov 03 22:45:47.329565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.329556 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\") pod \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " Nov 03 22:45:47.329775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.329586 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\") pod \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\" (UID: \"48d5c789-7603-4e49-b0f4-1d0835b6dc3c\") " Nov 03 22:45:47.330021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.329969 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca" pod "48d5c789-7603-4e49-b0f4-1d0835b6dc3c" (UID: "48d5c789-7603-4e49-b0f4-1d0835b6dc3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:47.330565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.330538 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.331861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.331834 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv" pod "48d5c789-7603-4e49-b0f4-1d0835b6dc3c" (UID: "48d5c789-7603-4e49-b0f4-1d0835b6dc3c"). InnerVolumeSpecName "kube-api-access-kkffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:47.331861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.331843 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca" pod "48d5c789-7603-4e49-b0f4-1d0835b6dc3c" (UID: "48d5c789-7603-4e49-b0f4-1d0835b6dc3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:47.433013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.432482 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkffv\" (UniqueName: \"kubernetes.io/projected/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-kube-api-access-kkffv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.433013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.432588 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d5c789-7603-4e49-b0f4-1d0835b6dc3c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:47.703466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.703431 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:47.705449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.705365 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:47.706341 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:47.706317 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:45:48.058119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:48.058038 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" Nov 03 22:45:48.058543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:48.058516 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b" event={"ID":"48d5c789-7603-4e49-b0f4-1d0835b6dc3c","Type":"ContainerDied","Data":"37794bf3ce5ada922874d1ca7772b057a016ae6e74d62735e5d66860c7ef5e0c"} Nov 03 22:45:48.058603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:45:48.058572 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37794bf3ce5ada922874d1ca7772b057a016ae6e74d62735e5d66860c7ef5e0c" Nov 03 22:45:51.625868 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:45:51.625832 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:46:02.617123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.617082 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 22:46:02.618060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.618038 5051 cpu_manager.go:414] "containerMap: removing container" podUID="48d5c789-7603-4e49-b0f4-1d0835b6dc3c" containerName="revoke-keys" Nov 03 22:46:02.618060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.618060 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d5c789-7603-4e49-b0f4-1d0835b6dc3c" containerName="revoke-keys" Nov 03 22:46:02.618256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.618242 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d5c789-7603-4e49-b0f4-1d0835b6dc3c" containerName="revoke-keys" Nov 03 22:46:02.622388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.622367 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.675825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.675786 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:46:02.675825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.675786 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:46:02.676117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.675824 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:46:02.676117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.675839 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-69szt\"" Nov 03 22:46:02.733246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.733206 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 22:46:02.791797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.791761 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792002 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44wt\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792044 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792070 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792129 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792169 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.792232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.792217 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894060 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44wt\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894105 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894123 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894166 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894192 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894223 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.894455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.894423 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.895144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.895117 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.897056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.897033 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.897182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.897075 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.897482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.897449 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.897692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.897674 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.955003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.954957 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:02.955370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:02.955344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44wt\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vndrh\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:03.237279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:03.237187 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:03.698739 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:03.698703 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823497da_9aa1_4fdd_a564_faf36bc4f1ad.slice/crio-9776a5b00d8d0a1e1ebfd8cf3dacf855993c36007b02e23429dd35da7c240bf4 WatchSource:0}: Error finding container 9776a5b00d8d0a1e1ebfd8cf3dacf855993c36007b02e23429dd35da7c240bf4: Status 404 returned error can't find the container with id 9776a5b00d8d0a1e1ebfd8cf3dacf855993c36007b02e23429dd35da7c240bf4 Nov 03 22:46:03.699213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:03.699115 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 22:46:03.701203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:03.701180 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:46:04.229077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:04.229041 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" event={"ID":"823497da-9aa1-4fdd-a564-faf36bc4f1ad","Type":"ContainerStarted","Data":"9776a5b00d8d0a1e1ebfd8cf3dacf855993c36007b02e23429dd35da7c240bf4"} Nov 03 22:46:06.561739 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.561705 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:46:06.567674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.567653 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.626872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.626835 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:46:06.654876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.654828 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.654980 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655097 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655148 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbdf\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655288 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655350 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655471 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655518 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.655677 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.655660 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.756920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.756887 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.756920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.756935 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757299 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757335 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757345 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbdf\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757416 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757441 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757500 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.757752 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.757657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.758056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.758029 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.760050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.760021 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.760291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.760272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.818594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.818521 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.818826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.818801 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbdf\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:06.884894 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:06.884858 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:07.348299 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:07.348262 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740d86ff_0522_4bf4_8ecc_89f0ce1457e7.slice/crio-10c94747f5b2e7b0f2baac7a9010aae9d45aa8fd5886721e09683ec4cefdab3c WatchSource:0}: Error finding container 10c94747f5b2e7b0f2baac7a9010aae9d45aa8fd5886721e09683ec4cefdab3c: Status 404 returned error can't find the container with id 10c94747f5b2e7b0f2baac7a9010aae9d45aa8fd5886721e09683ec4cefdab3c Nov 03 22:46:07.351830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.351790 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:07.351949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.351882 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:07.351949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.351931 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:07.354028 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.354001 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:46:07.939165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.939129 5051 scope.go:122] "RemoveContainer" containerID="7d12796f8af107b94b539dc33615274783e043f6928d33321f1f9cd2d760fcc3" Nov 03 22:46:07.953974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.953945 5051 scope.go:122] "RemoveContainer" containerID="03911032db8f9ba371a635082cd54c17e7b1138701368632ebf7516d4dbf193d" Nov 03 22:46:07.971084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.971051 5051 scope.go:122] "RemoveContainer" containerID="8533a15597b6262553fc6bcbe3b76b9fc4080f76d7ec0961ac3658e580250d0e" Nov 03 22:46:07.988157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:07.988133 5051 scope.go:122] "RemoveContainer" containerID="cf33a7cfd0838e5150ac6cac51506213816930aab5dc2bfa641a9f4963987a1d" Nov 03 22:46:08.009152 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.008935 5051 scope.go:122] "RemoveContainer" containerID="8616d22878f20e966e5ea8b5d21e0841ded4508f003f36eb96a98d7f63ec691e" Nov 03 22:46:08.025767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.025746 5051 scope.go:122] "RemoveContainer" containerID="9ee91d4116bc341dc90ff70f16fba4ee4ac2d09ae07401ae2a10bf1ecd42dd6d" Nov 03 22:46:08.310099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.309040 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" event={"ID":"740d86ff-0522-4bf4-8ecc-89f0ce1457e7","Type":"ContainerStarted","Data":"15abb30c357316f21974c3b2fa97c98ebc9604d9594e3acadf50a7a03d9c3cfc"} Nov 03 22:46:08.310099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.309155 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" event={"ID":"740d86ff-0522-4bf4-8ecc-89f0ce1457e7","Type":"ContainerStarted","Data":"10c94747f5b2e7b0f2baac7a9010aae9d45aa8fd5886721e09683ec4cefdab3c"} Nov 03 22:46:08.497783 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.497721 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podStartSLOduration=2.497704184 podStartE2EDuration="2.497704184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:08.486721984 +0000 UTC m=+1442.224066210" watchObservedRunningTime="2026-11-03 22:46:08.497704184 +0000 UTC m=+1442.235048409" Nov 03 22:46:08.888466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.888428 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:08.890688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.890645 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:08.890868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:08.890709 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:09.600406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.600365 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw"] Nov 03 22:46:09.613924 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.613889 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.718323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.718275 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw"] Nov 03 22:46:09.746167 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746129 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22tb\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-kube-api-access-n22tb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746197 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746412 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746671 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e4cfcd-d1db-4503-8a90-740070194e2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746713 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746754 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746735 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.746920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.746765 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.817076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.817033 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:09.817266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.817118 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:09.848371 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848339 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848451 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848624 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e4cfcd-d1db-4503-8a90-740070194e2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848650 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848670 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848701 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.848982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.848782 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22tb\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-kube-api-access-n22tb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.849591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.849563 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.849708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.849626 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.849779 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.849705 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.849840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.849824 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.850247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.850227 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e4cfcd-d1db-4503-8a90-740070194e2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.851845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.851802 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.851947 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.851901 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.890027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.886404 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:09.890027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.886483 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:09.909892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.909849 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.910095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.910011 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22tb\" (UniqueName: \"kubernetes.io/projected/23e4cfcd-d1db-4503-8a90-740070194e2b-kube-api-access-n22tb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zhbvw\" (UID: \"23e4cfcd-d1db-4503-8a90-740070194e2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:09.935823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:09.935785 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:10.334485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.334384 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" event={"ID":"823497da-9aa1-4fdd-a564-faf36bc4f1ad","Type":"ContainerStarted","Data":"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50"} Nov 03 22:46:10.334711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.334688 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:10.336460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.336434 5051 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vndrh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:46:10.336569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.336481 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:46:10.492061 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:10.491133 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e4cfcd_d1db_4503_8a90_740070194e2b.slice/crio-36fb8847751774a33240aca056760ef1d2ce86347077f8ee1d57f4d3cf36dc0a WatchSource:0}: Error finding container 36fb8847751774a33240aca056760ef1d2ce86347077f8ee1d57f4d3cf36dc0a: Status 404 returned error can't find the container with id 36fb8847751774a33240aca056760ef1d2ce86347077f8ee1d57f4d3cf36dc0a Nov 03 22:46:10.496854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.496806 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:10.496968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.496890 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:10.496968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.496938 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:10.502008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.501966 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw"] Nov 03 22:46:10.504557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.504503 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" podStartSLOduration=2.390397018 podStartE2EDuration="8.504487017s" totalImagesPullingTime="6.114089999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:10.455590478 +0000 UTC m=+1444.192934703" watchObservedRunningTime="2026-11-03 22:46:10.504487017 +0000 UTC m=+1444.241831244" Nov 03 22:46:10.886577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.886539 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:10.887238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:10.887208 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:11.896904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:11.896865 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:11.897608 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:11.897571 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:12.224335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.224233 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" event={"ID":"23e4cfcd-d1db-4503-8a90-740070194e2b","Type":"ContainerStarted","Data":"1933a4768f0da483e68f0366882a5e49e6878ed371efb7b62cc47d64e8062952"} Nov 03 22:46:12.224770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.224733 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" event={"ID":"23e4cfcd-d1db-4503-8a90-740070194e2b","Type":"ContainerStarted","Data":"36fb8847751774a33240aca056760ef1d2ce86347077f8ee1d57f4d3cf36dc0a"} Nov 03 22:46:12.225423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.225401 5051 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vndrh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:46:12.225622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.225602 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:46:12.265153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.265106 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:46:12.441015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.440090 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:46:12.550580 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.550320 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podStartSLOduration=3.550300323 podStartE2EDuration="3.550300323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:12.408953525 +0000 UTC m=+1446.146297752" watchObservedRunningTime="2026-11-03 22:46:12.550300323 +0000 UTC m=+1446.287644549" Nov 03 22:46:12.924019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.910748 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:46:12.924019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.912124 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:12.924019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.912192 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:12.939126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.939096 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:12.949019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.942102 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:12.949019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:12.942159 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:13.487018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:13.486113 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:46:13.487018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:13.486750 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:46:13.514832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:13.514800 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 22:46:13.893014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:13.888839 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:13.893014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:13.888901 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:14.044972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.036580 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:14.069706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.045573 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:14.898548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.898055 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:46:14.898548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.898138 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:46:14.937138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.937092 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:14.937354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:14.937181 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:15.239228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:15.239147 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 22:46:15.953477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:15.953415 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:15.953791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:15.953754 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:16.543670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:16.543641 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:16.544576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:16.544554 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:16.546612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:16.546590 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:46:16.939951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:16.939918 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:16.940177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:16.939974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:17.939017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:17.938370 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:17.939017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:17.938432 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:18.937056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:18.937024 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:18.937247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:18.937076 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:19.936586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:19.936550 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zhbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:46:19.937121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:19.936623 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" podUID="23e4cfcd-d1db-4503-8a90-740070194e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:46:21.405557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:21.405520 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:21.406229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:21.406201 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:21.407203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:21.407185 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zhbvw" Nov 03 22:46:21.626488 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:46:21.626453 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:46:24.688647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.688591 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7"] Nov 03 22:46:24.694097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.694061 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.819860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.819804 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.819884 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frp49\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-kube-api-access-frp49\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.819925 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820092 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820216 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820264 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820360 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0c86f-7342-49af-a069-ed9ff722f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820409 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.820594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.820497 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930407 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930603 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0c86f-7342-49af-a069-ed9ff722f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930658 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930755 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930895 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.930955 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frp49\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-kube-api-access-frp49\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.931024 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.933030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.931216 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.959075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.955525 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.959075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.955814 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.959075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.956458 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.959075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.956719 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:24.959075 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:24.957447 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b0c86f-7342-49af-a069-ed9ff722f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.024066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.023648 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.090731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.070415 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7"] Nov 03 22:46:25.090731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.074820 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.090731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.078261 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frp49\" (UniqueName: \"kubernetes.io/projected/f6b0c86f-7342-49af-a069-ed9ff722f54b-kube-api-access-frp49\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.090731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.079124 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c86f-7342-49af-a069-ed9ff722f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zkrw7\" (UID: \"f6b0c86f-7342-49af-a069-ed9ff722f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.316936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.316814 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:25.581773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.581657 5051 generic.go:356] "Generic (PLEG): container finished" podID="1573b327-16ea-4f19-832b-8559cdfec54b" containerID="e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0" exitCode=0 Nov 03 22:46:25.581773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.581736 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerDied","Data":"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0"} Nov 03 22:46:25.582358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.582338 5051 scope.go:122] "RemoveContainer" containerID="e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0" Nov 03 22:46:25.584398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.584353 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.584566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.584460 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.584566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.584514 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.863778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.851870 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.863778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.851982 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.863778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.852050 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.863778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:25.852845 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7"] Nov 03 22:46:26.619303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:26.619267 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerStarted","Data":"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac"} Nov 03 22:46:26.621270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:26.621235 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" event={"ID":"f6b0c86f-7342-49af-a069-ed9ff722f54b","Type":"ContainerStarted","Data":"363abc057d0f641ef2a5c22e8a0a37d8e3b0711f6f67fe4ca0951dc174424c66"} Nov 03 22:46:26.621496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:26.621290 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" event={"ID":"f6b0c86f-7342-49af-a069-ed9ff722f54b","Type":"ContainerStarted","Data":"ee428f24aedb08abc30c14baee6428d952cd52a91454d634363d248b4d8d0caa"} Nov 03 22:46:26.869026 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:26.866230 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podStartSLOduration=2.866211088 podStartE2EDuration="2.866211088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:26.865650325 +0000 UTC m=+1460.602994552" watchObservedRunningTime="2026-11-03 22:46:26.866211088 +0000 UTC m=+1460.603555334" Nov 03 22:46:27.320138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.320103 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:27.321249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.321218 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:27.321358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.321295 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:27.413049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.408428 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:46:27.413049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.411010 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:27.413049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.411095 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:27.541098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:27.540543 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:46:28.321021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:28.320546 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:28.321021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:28.320609 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:28.408018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:28.407748 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:28.408018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:28.407836 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:29.322086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:29.320832 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:29.322086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:29.320921 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:29.463020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:29.443548 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:29.463020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:29.443634 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:30.113298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.113244 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf"] Nov 03 22:46:30.134942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.134450 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.154615 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.154584 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf"] Nov 03 22:46:30.229020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.228337 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:30.229020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.228539 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:46:30.230339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.230313 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.230560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.230539 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827d4\" (UniqueName: \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.230622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.230582 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.321016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.320874 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:30.321016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.320948 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:30.332051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.332018 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.332556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.332298 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.332556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.332490 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827d4\" (UniqueName: \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.332788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.332763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.337634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.337607 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.425072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.409850 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:30.425072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.409910 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:30.425072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.421253 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827d4\" (UniqueName: \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\") pod \"maas-api-revoke-sub-36264199-lpxqf\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:30.452568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:30.452536 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:31.289092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.287295 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf"] Nov 03 22:46:31.337113 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.331651 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:31.390425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.386171 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:31.428119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.414475 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:31.428119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.414550 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:31.751580 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:31.751539 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" event={"ID":"8b1a3458-c9df-47ce-abf2-a2f8b0094092","Type":"ContainerStarted","Data":"c92f6ab23735de5e4d02e3fc98d4ee4d6e22cff7900f4742b134210620ca693b"} Nov 03 22:46:32.043296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.043246 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q"] Nov 03 22:46:32.073125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.059359 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.151020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.148583 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.151020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.148761 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8br\" (UniqueName: \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.151020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.149041 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.244053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.243924 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q"] Nov 03 22:46:32.323196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.321465 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:32.323196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.321616 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:32.323196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.322650 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.323196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.322790 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8br\" (UniqueName: \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.323196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.323062 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.323857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.323761 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.326737 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.326707 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.416081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.411081 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 22:46:32.416081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.411151 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 22:46:32.423826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.423786 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8br\" (UniqueName: \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\") pod \"maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.542778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.542505 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:32.772054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.771969 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" event={"ID":"8b1a3458-c9df-47ce-abf2-a2f8b0094092","Type":"ContainerStarted","Data":"524a35f6a9eaf13fddc2cda89de0f139badadc65352edad7570e68c641dce41a"} Nov 03 22:46:32.888921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:32.888603 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" podStartSLOduration=2.888584061 podStartE2EDuration="2.888584061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:32.888110375 +0000 UTC m=+1466.625454601" watchObservedRunningTime="2026-11-03 22:46:32.888584061 +0000 UTC m=+1466.625928287" Nov 03 22:46:33.008296 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:33.008262 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850a4b89_a106_4213_8a4f_0064de584ed7.slice/crio-e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e WatchSource:0}: Error finding container e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e: Status 404 returned error can't find the container with id e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e Nov 03 22:46:33.010033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.009963 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q"] Nov 03 22:46:33.317839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.317755 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:33.317839 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.317814 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:33.413278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.413247 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:46:33.414594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.414571 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:46:33.793014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.792481 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" event={"ID":"850a4b89-a106-4213-8a4f-0064de584ed7","Type":"ContainerStarted","Data":"2fb2c311643ff1ffb6580ef1f89fe73f5bc6b85f2189c344662dc8b62c4a9386"} Nov 03 22:46:33.793014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.792566 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" event={"ID":"850a4b89-a106-4213-8a4f-0064de584ed7","Type":"ContainerStarted","Data":"e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e"} Nov 03 22:46:33.795115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.795081 5051 generic.go:356] "Generic (PLEG): container finished" podID="8b1a3458-c9df-47ce-abf2-a2f8b0094092" containerID="524a35f6a9eaf13fddc2cda89de0f139badadc65352edad7570e68c641dce41a" exitCode=0 Nov 03 22:46:33.796435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.796404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" event={"ID":"8b1a3458-c9df-47ce-abf2-a2f8b0094092","Type":"ContainerDied","Data":"524a35f6a9eaf13fddc2cda89de0f139badadc65352edad7570e68c641dce41a"} Nov 03 22:46:33.908416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:33.908357 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" podStartSLOduration=2.90833891 podStartE2EDuration="2.90833891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:33.90715869 +0000 UTC m=+1467.644502916" watchObservedRunningTime="2026-11-03 22:46:33.90833891 +0000 UTC m=+1467.645683135" Nov 03 22:46:34.317567 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:34.317521 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:46:34.317794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:34.317588 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:46:34.815155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:34.815109 5051 generic.go:356] "Generic (PLEG): container finished" podID="850a4b89-a106-4213-8a4f-0064de584ed7" containerID="2fb2c311643ff1ffb6580ef1f89fe73f5bc6b85f2189c344662dc8b62c4a9386" exitCode=0 Nov 03 22:46:34.815620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:34.815249 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" event={"ID":"850a4b89-a106-4213-8a4f-0064de584ed7","Type":"ContainerDied","Data":"2fb2c311643ff1ffb6580ef1f89fe73f5bc6b85f2189c344662dc8b62c4a9386"} Nov 03 22:46:35.176565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.176533 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:35.223089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.222943 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\" (UniqueName: \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\") pod \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " Nov 03 22:46:35.223388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.223240 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\") pod \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " Nov 03 22:46:35.223495 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.223471 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\") pod \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\" (UID: \"8b1a3458-c9df-47ce-abf2-a2f8b0094092\") " Nov 03 22:46:35.224848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.224802 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca" pod "8b1a3458-c9df-47ce-abf2-a2f8b0094092" (UID: "8b1a3458-c9df-47ce-abf2-a2f8b0094092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:35.225258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.225238 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:35.231089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.231056 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca" pod "8b1a3458-c9df-47ce-abf2-a2f8b0094092" (UID: "8b1a3458-c9df-47ce-abf2-a2f8b0094092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:35.235071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.234979 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4" pod "8b1a3458-c9df-47ce-abf2-a2f8b0094092" (UID: "8b1a3458-c9df-47ce-abf2-a2f8b0094092"). InnerVolumeSpecName "kube-api-access-827d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:35.328426 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.328061 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827d4\" (UniqueName: \"kubernetes.io/projected/8b1a3458-c9df-47ce-abf2-a2f8b0094092-kube-api-access-827d4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:35.347563 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.347526 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a3458-c9df-47ce-abf2-a2f8b0094092-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:35.696346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.696312 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:35.697339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.697231 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:35.698896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.698285 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" Nov 03 22:46:35.826911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.826885 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" Nov 03 22:46:35.827418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.826920 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf" event={"ID":"8b1a3458-c9df-47ce-abf2-a2f8b0094092","Type":"ContainerDied","Data":"c92f6ab23735de5e4d02e3fc98d4ee4d6e22cff7900f4742b134210620ca693b"} Nov 03 22:46:35.827418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:35.826964 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92f6ab23735de5e4d02e3fc98d4ee4d6e22cff7900f4742b134210620ca693b" Nov 03 22:46:36.061320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.061285 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:36.167173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.166061 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\") pod \"850a4b89-a106-4213-8a4f-0064de584ed7\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " Nov 03 22:46:36.167173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.166214 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\") pod \"850a4b89-a106-4213-8a4f-0064de584ed7\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " Nov 03 22:46:36.167173 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.166567 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\" (UniqueName: \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\") pod \"850a4b89-a106-4213-8a4f-0064de584ed7\" (UID: \"850a4b89-a106-4213-8a4f-0064de584ed7\") " Nov 03 22:46:36.168224 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.168197 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca" pod "850a4b89-a106-4213-8a4f-0064de584ed7" (UID: "850a4b89-a106-4213-8a4f-0064de584ed7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:36.171605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.171572 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca" pod "850a4b89-a106-4213-8a4f-0064de584ed7" (UID: "850a4b89-a106-4213-8a4f-0064de584ed7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:36.171790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.171761 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br" pod "850a4b89-a106-4213-8a4f-0064de584ed7" (UID: "850a4b89-a106-4213-8a4f-0064de584ed7"). InnerVolumeSpecName "kube-api-access-4d8br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:36.269077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.268777 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8br\" (UniqueName: \"kubernetes.io/projected/850a4b89-a106-4213-8a4f-0064de584ed7-kube-api-access-4d8br\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:36.269077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.268823 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:36.269077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.268834 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a4b89-a106-4213-8a4f-0064de584ed7-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:36.840201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.840164 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" Nov 03 22:46:36.840201 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.840178 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q" event={"ID":"850a4b89-a106-4213-8a4f-0064de584ed7","Type":"ContainerDied","Data":"e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e"} Nov 03 22:46:36.840686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:36.840233 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9350ebfe11aa945f024af8b606bfaf3731beefa0d81affab5f665a6c37f6e4e" Nov 03 22:46:37.223300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.223234 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj"] Nov 03 22:46:37.224577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224528 5051 cpu_manager.go:414] "containerMap: removing container" podUID="850a4b89-a106-4213-8a4f-0064de584ed7" containerName="revoke-keys" Nov 03 22:46:37.224577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224580 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850a4b89-a106-4213-8a4f-0064de584ed7" containerName="revoke-keys" Nov 03 22:46:37.224696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224625 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8b1a3458-c9df-47ce-abf2-a2f8b0094092" containerName="revoke-keys" Nov 03 22:46:37.224696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224634 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1a3458-c9df-47ce-abf2-a2f8b0094092" containerName="revoke-keys" Nov 03 22:46:37.224930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224913 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="850a4b89-a106-4213-8a4f-0064de584ed7" containerName="revoke-keys" Nov 03 22:46:37.224965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.224935 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1a3458-c9df-47ce-abf2-a2f8b0094092" containerName="revoke-keys" Nov 03 22:46:37.230381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.230340 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.282068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.281946 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj"] Nov 03 22:46:37.323213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.323164 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:37.323213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.323209 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:46:37.390753 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.390710 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.390967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.390776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.391068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.391044 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h72h\" (UniqueName: \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.492713 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.492594 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h72h\" (UniqueName: \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.493085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.493061 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.493185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.493137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.493890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.493840 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.495916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.495889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.554205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.554167 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h72h\" (UniqueName: \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\") pod \"maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:37.849175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:37.849093 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:38.239244 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:38.239203 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be5927b_4b13_42a2_9b16_cdbb5b848d27.slice/crio-2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce WatchSource:0}: Error finding container 2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce: Status 404 returned error can't find the container with id 2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce Nov 03 22:46:38.240139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:38.240117 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj"] Nov 03 22:46:38.871846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:38.871787 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" event={"ID":"6be5927b-4b13-42a2-9b16-cdbb5b848d27","Type":"ContainerStarted","Data":"642795a4e574d8396b03ad7cd41d64280c415896a0f8fca6429dccf52fee8d01"} Nov 03 22:46:38.872327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:38.871901 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" event={"ID":"6be5927b-4b13-42a2-9b16-cdbb5b848d27","Type":"ContainerStarted","Data":"2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce"} Nov 03 22:46:38.986840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:38.986751 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" podStartSLOduration=1.9867299200000001 podStartE2EDuration="1.98672992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:38.986043501 +0000 UTC m=+1472.723387726" watchObservedRunningTime="2026-11-03 22:46:38.98672992 +0000 UTC m=+1472.724074145" Nov 03 22:46:39.891466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:39.891422 5051 generic.go:356] "Generic (PLEG): container finished" podID="6be5927b-4b13-42a2-9b16-cdbb5b848d27" containerID="642795a4e574d8396b03ad7cd41d64280c415896a0f8fca6429dccf52fee8d01" exitCode=0 Nov 03 22:46:39.892074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:39.891510 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" event={"ID":"6be5927b-4b13-42a2-9b16-cdbb5b848d27","Type":"ContainerDied","Data":"642795a4e574d8396b03ad7cd41d64280c415896a0f8fca6429dccf52fee8d01"} Nov 03 22:46:41.094753 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.094726 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:41.267235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.267117 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\") pod \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " Nov 03 22:46:41.267448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.267343 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\") pod \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " Nov 03 22:46:41.283376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.267525 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\" (UniqueName: \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\") pod \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\" (UID: \"6be5927b-4b13-42a2-9b16-cdbb5b848d27\") " Nov 03 22:46:41.291074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.291037 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca" pod "6be5927b-4b13-42a2-9b16-cdbb5b848d27" (UID: "6be5927b-4b13-42a2-9b16-cdbb5b848d27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:41.298935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.298896 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca" pod "6be5927b-4b13-42a2-9b16-cdbb5b848d27" (UID: "6be5927b-4b13-42a2-9b16-cdbb5b848d27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:41.303079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.302975 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h" pod "6be5927b-4b13-42a2-9b16-cdbb5b848d27" (UID: "6be5927b-4b13-42a2-9b16-cdbb5b848d27"). InnerVolumeSpecName "kube-api-access-4h72h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:41.386357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.386312 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:41.386357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.386344 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h72h\" (UniqueName: \"kubernetes.io/projected/6be5927b-4b13-42a2-9b16-cdbb5b848d27-kube-api-access-4h72h\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:41.386357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.386356 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be5927b-4b13-42a2-9b16-cdbb5b848d27-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:41.915134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.915083 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" event={"ID":"6be5927b-4b13-42a2-9b16-cdbb5b848d27","Type":"ContainerDied","Data":"2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce"} Nov 03 22:46:41.915134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.915128 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj" Nov 03 22:46:41.915378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:41.915156 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3f88013919df5183c18c31391cd777c052f879e23d368ae90d04559b0b33ce" Nov 03 22:46:49.020236 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.020141 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:46:49.131662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.131621 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:46:49.209398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.209359 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6"] Nov 03 22:46:49.210363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.210335 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6be5927b-4b13-42a2-9b16-cdbb5b848d27" containerName="revoke-keys" Nov 03 22:46:49.210363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.210362 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be5927b-4b13-42a2-9b16-cdbb5b848d27" containerName="revoke-keys" Nov 03 22:46:49.210585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.210541 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be5927b-4b13-42a2-9b16-cdbb5b848d27" containerName="revoke-keys" Nov 03 22:46:49.215011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.214966 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.273718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.269196 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:49.273718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.269343 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:46:49.273718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.273688 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6"] Nov 03 22:46:49.287204 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.287150 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.287455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.287329 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.287455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.287359 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn468\" (UniqueName: \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.389383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.389330 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.389631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.389502 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.389631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.389532 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn468\" (UniqueName: \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.391543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.391502 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.398786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.398739 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.456502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.456351 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn468\" (UniqueName: \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\") pod \"maas-api-revoke-sub-3f2cfc60-8r5k6\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:49.551763 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:49.545344 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:50.033252 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:46:50.033215 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bee6c4_58c4_41d1_92c0_1215b5a1805e.slice/crio-ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489 WatchSource:0}: Error finding container ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489: Status 404 returned error can't find the container with id ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489 Nov 03 22:46:50.033797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:50.033768 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6"] Nov 03 22:46:51.033071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:51.030200 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" event={"ID":"36bee6c4-58c4-41d1-92c0-1215b5a1805e","Type":"ContainerStarted","Data":"5d7bd00f634713909e658e861b230485c3e208a3cf95b8847e7c766fabd53b19"} Nov 03 22:46:51.033071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:51.030276 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" event={"ID":"36bee6c4-58c4-41d1-92c0-1215b5a1805e","Type":"ContainerStarted","Data":"ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489"} Nov 03 22:46:51.181587 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:51.179163 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" podStartSLOduration=2.179138039 podStartE2EDuration="2.179138039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:51.159713019 +0000 UTC m=+1484.897057244" watchObservedRunningTime="2026-11-03 22:46:51.179138039 +0000 UTC m=+1484.916482265" Nov 03 22:46:51.627409 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:46:51.627369 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:46:52.042321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:52.042208 5051 generic.go:356] "Generic (PLEG): container finished" podID="36bee6c4-58c4-41d1-92c0-1215b5a1805e" containerID="5d7bd00f634713909e658e861b230485c3e208a3cf95b8847e7c766fabd53b19" exitCode=0 Nov 03 22:46:52.042321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:52.042290 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" event={"ID":"36bee6c4-58c4-41d1-92c0-1215b5a1805e","Type":"ContainerDied","Data":"5d7bd00f634713909e658e861b230485c3e208a3cf95b8847e7c766fabd53b19"} Nov 03 22:46:53.259069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.259015 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:53.379710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.379669 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\" (UniqueName: \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\") pod \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " Nov 03 22:46:53.380280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.380254 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\") pod \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " Nov 03 22:46:53.380420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.380381 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\") pod \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\" (UID: \"36bee6c4-58c4-41d1-92c0-1215b5a1805e\") " Nov 03 22:46:53.381337 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.381292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca" pod "36bee6c4-58c4-41d1-92c0-1215b5a1805e" (UID: "36bee6c4-58c4-41d1-92c0-1215b5a1805e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:53.381735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.381715 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:53.383139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.383109 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468" pod "36bee6c4-58c4-41d1-92c0-1215b5a1805e" (UID: "36bee6c4-58c4-41d1-92c0-1215b5a1805e"). InnerVolumeSpecName "kube-api-access-nn468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:53.385537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.385502 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca" pod "36bee6c4-58c4-41d1-92c0-1215b5a1805e" (UID: "36bee6c4-58c4-41d1-92c0-1215b5a1805e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:53.483800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.483767 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn468\" (UniqueName: \"kubernetes.io/projected/36bee6c4-58c4-41d1-92c0-1215b5a1805e-kube-api-access-nn468\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:53.483800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:53.483798 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bee6c4-58c4-41d1-92c0-1215b5a1805e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:54.020836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.020766 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" podUID="386c549f-29ec-43eb-bdb5-81177026e39f" containerName="payload-processing" containerID="cri-o://51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b" gracePeriod=25 Nov 03 22:46:54.062938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.062904 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" Nov 03 22:46:54.063177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.062935 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6" event={"ID":"36bee6c4-58c4-41d1-92c0-1215b5a1805e","Type":"ContainerDied","Data":"ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489"} Nov 03 22:46:54.063177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.063040 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef51262b2194fd720945073e4f05c2685dde4d4df802c1b70c186a9a2f3d9489" Nov 03 22:46:54.132185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.132139 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" podUID="03571ac3-ce57-4961-95d6-8aa8259db43a" containerName="payload-pre-processing" containerID="cri-o://21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f" gracePeriod=25 Nov 03 22:46:54.388539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.380574 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:46:54.508563 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.508528 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\" (UniqueName: \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\") pod \"386c549f-29ec-43eb-bdb5-81177026e39f\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " Nov 03 22:46:54.508826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.508632 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\") pod \"386c549f-29ec-43eb-bdb5-81177026e39f\" (UID: \"386c549f-29ec-43eb-bdb5-81177026e39f\") " Nov 03 22:46:54.510542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.509695 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume" pod "386c549f-29ec-43eb-bdb5-81177026e39f" (UID: "386c549f-29ec-43eb-bdb5-81177026e39f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:54.511877 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.511840 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:46:54.512079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.511920 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs" pod "386c549f-29ec-43eb-bdb5-81177026e39f" (UID: "386c549f-29ec-43eb-bdb5-81177026e39f"). InnerVolumeSpecName "kube-api-access-gdffs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:54.610917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.610876 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\" (UniqueName: \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\") pod \"03571ac3-ce57-4961-95d6-8aa8259db43a\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " Nov 03 22:46:54.611183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.610971 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\") pod \"03571ac3-ce57-4961-95d6-8aa8259db43a\" (UID: \"03571ac3-ce57-4961-95d6-8aa8259db43a\") " Nov 03 22:46:54.611445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.611419 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume" pod "03571ac3-ce57-4961-95d6-8aa8259db43a" (UID: "03571ac3-ce57-4961-95d6-8aa8259db43a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:54.611906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.611882 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03571ac3-ce57-4961-95d6-8aa8259db43a-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:54.611948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.611912 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdffs\" (UniqueName: \"kubernetes.io/projected/386c549f-29ec-43eb-bdb5-81177026e39f-kube-api-access-gdffs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:54.611948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.611923 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386c549f-29ec-43eb-bdb5-81177026e39f-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:54.613841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.613811 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p" pod "03571ac3-ce57-4961-95d6-8aa8259db43a" (UID: "03571ac3-ce57-4961-95d6-8aa8259db43a"). InnerVolumeSpecName "kube-api-access-qj46p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:54.714274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:54.714222 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj46p\" (UniqueName: \"kubernetes.io/projected/03571ac3-ce57-4961-95d6-8aa8259db43a-kube-api-access-qj46p\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:46:55.078983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.078881 5051 generic.go:356] "Generic (PLEG): container finished" podID="386c549f-29ec-43eb-bdb5-81177026e39f" containerID="51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b" exitCode=0 Nov 03 22:46:55.078983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.078955 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" Nov 03 22:46:55.079503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.078967 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" event={"ID":"386c549f-29ec-43eb-bdb5-81177026e39f","Type":"ContainerDied","Data":"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b"} Nov 03 22:46:55.079627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.079549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj" event={"ID":"386c549f-29ec-43eb-bdb5-81177026e39f","Type":"ContainerDied","Data":"5c22d930c7ed9664df825a2e0572a18f7495edd2d4afd2ca058f627486549558"} Nov 03 22:46:55.079627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.079588 5051 scope.go:122] "RemoveContainer" containerID="51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b" Nov 03 22:46:55.086871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.086830 5051 generic.go:356] "Generic (PLEG): container finished" podID="03571ac3-ce57-4961-95d6-8aa8259db43a" containerID="21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f" exitCode=0 Nov 03 22:46:55.087054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.086957 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" Nov 03 22:46:55.087130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.087067 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" event={"ID":"03571ac3-ce57-4961-95d6-8aa8259db43a","Type":"ContainerDied","Data":"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f"} Nov 03 22:46:55.087130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.087121 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt" event={"ID":"03571ac3-ce57-4961-95d6-8aa8259db43a","Type":"ContainerDied","Data":"323a5d7b20f6daddeace756705ab10fb8e977ca7a6c70434732d9c76bb28ebba"} Nov 03 22:46:55.096261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.096108 5051 scope.go:122] "RemoveContainer" containerID="51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b" Nov 03 22:46:55.096593 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:46:55.096563 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b\": container with ID starting with 51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b not found: ID does not exist" containerID="51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b" Nov 03 22:46:55.096687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.096608 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b"} err="failed to get container status \"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b\": rpc error: code = NotFound desc = could not find container \"51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b\": container with ID starting with 51a9fa85e274707b2efb9fcb10bff238d159d91ccb2e106f5f0f22df3af5592b not found: ID does not exist" Nov 03 22:46:55.096687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.096633 5051 scope.go:122] "RemoveContainer" containerID="21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f" Nov 03 22:46:55.111742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.111715 5051 scope.go:122] "RemoveContainer" containerID="21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f" Nov 03 22:46:55.112135 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:46:55.112112 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f\": container with ID starting with 21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f not found: ID does not exist" containerID="21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f" Nov 03 22:46:55.112226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.112147 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f"} err="failed to get container status \"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f\": rpc error: code = NotFound desc = could not find container \"21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f\": container with ID starting with 21b7ce625c2b792934a9b872b25dfbee0cf8b7f8efb0bb67daf2239cd3f4cd7f not found: ID does not exist" Nov 03 22:46:55.258735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.258694 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:46:55.307448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.307416 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc129ccf-75b6c8f7d7-qtmzt"] Nov 03 22:46:55.369652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.369613 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:46:55.380062 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:55.380018 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc129ccf-9f4d47cd6-bv8bj"] Nov 03 22:46:56.414081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:56.414049 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03571ac3-ce57-4961-95d6-8aa8259db43a" path="/var/lib/kubelet/pods/03571ac3-ce57-4961-95d6-8aa8259db43a/volumes" Nov 03 22:46:56.414606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:46:56.414582 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386c549f-29ec-43eb-bdb5-81177026e39f" path="/var/lib/kubelet/pods/386c549f-29ec-43eb-bdb5-81177026e39f/volumes" Nov 03 22:47:02.738185 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.738144 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:02.740100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740077 5051 cpu_manager.go:414] "containerMap: removing container" podUID="386c549f-29ec-43eb-bdb5-81177026e39f" containerName="payload-processing" Nov 03 22:47:02.740237 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740227 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386c549f-29ec-43eb-bdb5-81177026e39f" containerName="payload-processing" Nov 03 22:47:02.740304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740297 5051 cpu_manager.go:414] "containerMap: removing container" podUID="36bee6c4-58c4-41d1-92c0-1215b5a1805e" containerName="revoke-keys" Nov 03 22:47:02.740353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740347 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bee6c4-58c4-41d1-92c0-1215b5a1805e" containerName="revoke-keys" Nov 03 22:47:02.740429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740422 5051 cpu_manager.go:414] "containerMap: removing container" podUID="03571ac3-ce57-4961-95d6-8aa8259db43a" containerName="payload-pre-processing" Nov 03 22:47:02.740477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740470 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03571ac3-ce57-4961-95d6-8aa8259db43a" containerName="payload-pre-processing" Nov 03 22:47:02.740689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740680 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="386c549f-29ec-43eb-bdb5-81177026e39f" containerName="payload-processing" Nov 03 22:47:02.740744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740738 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bee6c4-58c4-41d1-92c0-1215b5a1805e" containerName="revoke-keys" Nov 03 22:47:02.740802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.740795 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="03571ac3-ce57-4961-95d6-8aa8259db43a" containerName="payload-pre-processing" Nov 03 22:47:02.746569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.746538 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.752193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.752153 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:47:02.752351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.752232 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:02.803922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.803886 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fc129ccf-gw-openshift-default-dockercfg-fjr2t\"" Nov 03 22:47:02.852690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.852643 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.853083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.853053 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkxg\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.853622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.853597 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.853808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.853790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.854105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.854085 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.854313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.854295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.854519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.854502 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.854703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.854686 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.854876 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.854857 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.890016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.876068 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:02.993078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.992700 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.993078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.992809 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.993078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.992837 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.993078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993018 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:02.993078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993053 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkxg\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993332 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993401 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993431 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.993970 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.995353 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.995583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.996224 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:02.997319 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.018449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.007273 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.072017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.065164 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.084316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.084241 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkxg\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.116280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.115602 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.377209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.377158 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:03.832212 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:47:03.832165 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd785f797_6b89_4d08_8d83_164a159ccdbb.slice/crio-2040c92c3f46e685b90980e3479e98aa99b9292a3e3fa4c42f8b8fe2d3d172df WatchSource:0}: Error finding container 2040c92c3f46e685b90980e3479e98aa99b9292a3e3fa4c42f8b8fe2d3d172df: Status 404 returned error can't find the container with id 2040c92c3f46e685b90980e3479e98aa99b9292a3e3fa4c42f8b8fe2d3d172df Nov 03 22:47:03.833388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.833359 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:03.834902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.834863 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:03.835122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.834963 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:03.835122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:03.835058 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:04.058088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.058046 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n"] Nov 03 22:47:04.063832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.063788 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07e0f07d-rct6n"] Nov 03 22:47:04.280788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.279968 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" event={"ID":"d785f797-6b89-4d08-8d83-164a159ccdbb","Type":"ContainerStarted","Data":"9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1"} Nov 03 22:47:04.280788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.280092 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" event={"ID":"d785f797-6b89-4d08-8d83-164a159ccdbb","Type":"ContainerStarted","Data":"2040c92c3f46e685b90980e3479e98aa99b9292a3e3fa4c42f8b8fe2d3d172df"} Nov 03 22:47:04.377391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.377346 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:04.380357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.380321 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:04.380571 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.380389 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:04.404899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.404823 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podStartSLOduration=2.404800412 podStartE2EDuration="2.404800412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:04.401403052 +0000 UTC m=+1498.138747315" watchObservedRunningTime="2026-11-03 22:47:04.404800412 +0000 UTC m=+1498.142144638" Nov 03 22:47:04.414850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:04.414809 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a709e5-55e0-445e-a974-c368be486346" path="/var/lib/kubelet/pods/70a709e5-55e0-445e-a974-c368be486346/volumes" Nov 03 22:47:05.383520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:05.383471 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:05.384103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:05.383547 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:06.381398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:06.381341 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:06.381781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:06.381453 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:06.737804 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:06.737720 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:47:06.737804 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:06.737770 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:47:07.379018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:07.378487 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:07.379018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:07.378569 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:08.383022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:08.379827 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:08.383022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:08.379897 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:09.383029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:09.377782 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 22:47:09.383029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:09.377866 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 22:47:10.383733 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:10.383695 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:10.384317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:10.384291 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:10.385092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:10.385072 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:16.734624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:16.734591 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:47:16.735068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:16.734804 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" containerID="cri-o://0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac" gracePeriod=30 Nov 03 22:47:16.746089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:16.746044 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q"] Nov 03 22:47:16.752518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:16.752480 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-468d8d1b-8vf6q"] Nov 03 22:47:18.416286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:18.416246 5051 patch_prober.go:29] interesting pod/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 22:47:18.425124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:18.416330 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:18.428214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:18.427536 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850a4b89-a106-4213-8a4f-0064de584ed7" path="/var/lib/kubelet/pods/850a4b89-a106-4213-8a4f-0064de584ed7/volumes" Nov 03 22:47:21.627780 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:47:21.627741 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:47:22.071340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.071293 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201331 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201527 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201611 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201685 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201738 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201863 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.201949 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.202011 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.203081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.202071 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\") pod \"1573b327-16ea-4f19-832b-8559cdfec54b\" (UID: \"1573b327-16ea-4f19-832b-8559cdfec54b\") " Nov 03 22:47:22.205760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.205718 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:22.206391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.206359 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:22.222731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.207684 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:22.223151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.223089 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:22.223459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.223427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:22.273243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.269236 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:22.273243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.269839 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:22.273243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.269916 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:22.273243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.270036 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z" pod "1573b327-16ea-4f19-832b-8559cdfec54b" (UID: "1573b327-16ea-4f19-832b-8559cdfec54b"). InnerVolumeSpecName "kube-api-access-89h7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310238 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89h7z\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-kube-api-access-89h7z\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310279 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310294 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1573b327-16ea-4f19-832b-8559cdfec54b-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310307 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310320 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310335 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1573b327-16ea-4f19-832b-8559cdfec54b-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310347 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310360 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1573b327-16ea-4f19-832b-8559cdfec54b-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.311021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.310372 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1573b327-16ea-4f19-832b-8559cdfec54b-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:22.516836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.512439 5051 generic.go:356] "Generic (PLEG): container finished" podID="1573b327-16ea-4f19-832b-8559cdfec54b" containerID="0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac" exitCode=0 Nov 03 22:47:22.516836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.512514 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" Nov 03 22:47:22.516836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.512944 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerDied","Data":"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac"} Nov 03 22:47:22.516836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.513037 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8" event={"ID":"1573b327-16ea-4f19-832b-8559cdfec54b","Type":"ContainerDied","Data":"1dc5076740ff1f03ce401298fa2f962029dd3d42c712de8f25d6c36336356bde"} Nov 03 22:47:22.516836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.513076 5051 scope.go:122] "RemoveContainer" containerID="0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac" Nov 03 22:47:22.546499 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.546471 5051 scope.go:122] "RemoveContainer" containerID="e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0" Nov 03 22:47:22.561251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.561223 5051 scope.go:122] "RemoveContainer" containerID="0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac" Nov 03 22:47:22.561678 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:47:22.561652 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac\": container with ID starting with 0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac not found: ID does not exist" containerID="0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac" Nov 03 22:47:22.561781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.561692 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac"} err="failed to get container status \"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac\": rpc error: code = NotFound desc = could not find container \"0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac\": container with ID starting with 0cca8a31932af4f84f54d927c4f829005523ab2cb3e1ee4c4a21ef12a23a64ac not found: ID does not exist" Nov 03 22:47:22.561781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.561720 5051 scope.go:122] "RemoveContainer" containerID="e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0" Nov 03 22:47:22.562459 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:47:22.562431 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0\": container with ID starting with e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0 not found: ID does not exist" containerID="e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0" Nov 03 22:47:22.562538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.562466 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0"} err="failed to get container status \"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0\": rpc error: code = NotFound desc = could not find container \"e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0\": container with ID starting with e1f2444884236b77b03a2f25d0e55c4df0295a9146e3f8038232f1895eef97d0 not found: ID does not exist" Nov 03 22:47:22.684738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.684689 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:47:22.691667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:22.691631 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-468d8d1b-openshift-default-685f7b56f5-rkfh8"] Nov 03 22:47:23.112450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:23.112412 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj"] Nov 03 22:47:23.118926 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:23.118880 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc129ccf-8b6vj"] Nov 03 22:47:23.187363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:23.186699 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:23.187363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:23.186960 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" containerID="cri-o://9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1" gracePeriod=30 Nov 03 22:47:24.415745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:24.415703 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" path="/var/lib/kubelet/pods/1573b327-16ea-4f19-832b-8559cdfec54b/volumes" Nov 03 22:47:24.416517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:24.416491 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be5927b-4b13-42a2-9b16-cdbb5b848d27" path="/var/lib/kubelet/pods/6be5927b-4b13-42a2-9b16-cdbb5b848d27/volumes" Nov 03 22:47:25.432022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:25.430551 5051 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:25.432022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:25.430692 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:28.280273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.280199 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x"] Nov 03 22:47:28.310050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.292664 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a9ec0a-ffj6x"] Nov 03 22:47:28.488603 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:47:28.488538 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd785f797_6b89_4d08_8d83_164a159ccdbb.slice/crio-conmon-9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:47:28.510618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.510570 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4fcf00-89e4-45e8-95b1-0e272580b7ca" path="/var/lib/kubelet/pods/ce4fcf00-89e4-45e8-95b1-0e272580b7ca/volumes" Nov 03 22:47:28.835086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.835054 5051 generic.go:356] "Generic (PLEG): container finished" podID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerID="9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1" exitCode=0 Nov 03 22:47:28.835283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.835126 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" event={"ID":"d785f797-6b89-4d08-8d83-164a159ccdbb","Type":"ContainerDied","Data":"9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1"} Nov 03 22:47:28.859227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.859185 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:28.946622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.946577 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.946855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.946640 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.946855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.946750 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.946978 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.947131 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.947196 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.947250 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.947281 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.947521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.947359 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\") pod \"d785f797-6b89-4d08-8d83-164a159ccdbb\" (UID: \"d785f797-6b89-4d08-8d83-164a159ccdbb\") " Nov 03 22:47:28.948505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.948474 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:28.948724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.948702 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:28.949413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.949368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:28.949532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.949470 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:28.949748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.949719 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:28.952105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.952069 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:28.952238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.952106 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:28.952304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.952294 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:28.952599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:28.952581 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg" pod "d785f797-6b89-4d08-8d83-164a159ccdbb" (UID: "d785f797-6b89-4d08-8d83-164a159ccdbb"). InnerVolumeSpecName "kube-api-access-qzkxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:29.050388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050352 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050385 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050394 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d785f797-6b89-4d08-8d83-164a159ccdbb-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050403 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050414 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzkxg\" (UniqueName: \"kubernetes.io/projected/d785f797-6b89-4d08-8d83-164a159ccdbb-kube-api-access-qzkxg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050423 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050431 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050442 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.050636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.050451 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d785f797-6b89-4d08-8d83-164a159ccdbb-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.846913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.846884 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" Nov 03 22:47:29.847445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.846926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx" event={"ID":"d785f797-6b89-4d08-8d83-164a159ccdbb","Type":"ContainerDied","Data":"2040c92c3f46e685b90980e3479e98aa99b9292a3e3fa4c42f8b8fe2d3d172df"} Nov 03 22:47:29.847445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:29.847025 5051 scope.go:122] "RemoveContainer" containerID="9f65526a40a003eabcd10a375931b2ca690e29af2d308f0b6e7b920cdf85dbf1" Nov 03 22:47:30.065851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:30.065815 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:30.127322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:30.117962 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-8hmfx"] Nov 03 22:47:30.411012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:30.410895 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" path="/var/lib/kubelet/pods/d785f797-6b89-4d08-8d83-164a159ccdbb/volumes" Nov 03 22:47:39.050233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:39.050193 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9"] Nov 03 22:47:39.099576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:39.099537 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bcb65e-rgww9"] Nov 03 22:47:40.412302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:40.412259 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e6a0a0-b6df-4080-a98c-baad0b934992" path="/var/lib/kubelet/pods/a3e6a0a0-b6df-4080-a98c-baad0b934992/volumes" Nov 03 22:47:51.629405 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:47:51.629367 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:47:57.059741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.059694 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk"] Nov 03 22:47:57.109273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.109234 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67bd5d5-qtddk"] Nov 03 22:47:57.782789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.782746 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks"] Nov 03 22:47:57.784396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784356 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.784634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784611 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.784790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784691 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" Nov 03 22:47:57.784790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784700 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" Nov 03 22:47:57.784938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784925 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d785f797-6b89-4d08-8d83-164a159ccdbb" containerName="istio-proxy" Nov 03 22:47:57.785069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.784946 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.800280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.798446 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.845017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.841742 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.845017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.841790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr8c\" (UniqueName: \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.845017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.842144 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.874189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.872471 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:47:57.874189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.873212 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:57.874189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.873256 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks"] Nov 03 22:47:57.945654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.945619 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 22:47:57.947261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.947230 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.947532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.947515 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.947957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.947937 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1573b327-16ea-4f19-832b-8559cdfec54b" containerName="istio-proxy" Nov 03 22:47:57.954123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.953723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.954123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.954040 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.954123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.954078 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr8c\" (UniqueName: \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.958788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.958235 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:57.991123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:57.991086 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.012771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.012731 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 22:47:58.055033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.049045 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c93e70\"" Nov 03 22:47:58.055033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.049303 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c93e70-dockercfg-469w7\"" Nov 03 22:47:58.067402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.067336 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:58.108385 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.108256 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 22:47:58.130044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.127388 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.130044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.127260 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 22:47:58.148878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.148834 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6lx\" (UniqueName: \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.149685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.149644 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.207049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.192800 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr8c\" (UniqueName: \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\") pod \"maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:58.265178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.265115 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6lx\" (UniqueName: \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.265729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.265694 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.270301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.270247 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.270731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.270689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.271917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.271889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.370279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.370245 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6lx\" (UniqueName: \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\") pod \"payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.375801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.372114 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.375801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.372456 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.375801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.373830 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.380892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.380851 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:58.414755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.414719 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9a2ff2-95f5-47e4-9941-e2f09ac011f0" path="/var/lib/kubelet/pods/dd9a2ff2-95f5-47e4-9941-e2f09ac011f0/volumes" Nov 03 22:47:58.446487 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.446433 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\") pod \"payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:58.452488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.452167 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:47:58.606056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:58.605896 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:59.010928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.010832 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 22:47:59.030525 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:47:59.030486 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24fb1fc_c5c1_4e0d_8b65_408f2e9e5b7c.slice/crio-3baf19adf109bd367217e167fa63fdeda380bc64584b63b866437a72798f7c56 WatchSource:0}: Error finding container 3baf19adf109bd367217e167fa63fdeda380bc64584b63b866437a72798f7c56: Status 404 returned error can't find the container with id 3baf19adf109bd367217e167fa63fdeda380bc64584b63b866437a72798f7c56 Nov 03 22:47:59.063710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.063665 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 22:47:59.064960 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:47:59.064923 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206aad65_1e1b_455e_bb78_383be74653af.slice/crio-4572d2d50f5dce92b394769f27f9d4476e36e2fa81cd1477d4ae84a5b3101620 WatchSource:0}: Error finding container 4572d2d50f5dce92b394769f27f9d4476e36e2fa81cd1477d4ae84a5b3101620: Status 404 returned error can't find the container with id 4572d2d50f5dce92b394769f27f9d4476e36e2fa81cd1477d4ae84a5b3101620 Nov 03 22:47:59.074755 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:47:59.074708 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8c4c0e_ee9a_4b87_abd2_7b0483f41886.slice/crio-71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099 WatchSource:0}: Error finding container 71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099: Status 404 returned error can't find the container with id 71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099 Nov 03 22:47:59.113361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.113318 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks"] Nov 03 22:47:59.315479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.315373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" event={"ID":"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886","Type":"ContainerStarted","Data":"71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099"} Nov 03 22:47:59.317406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.317366 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" event={"ID":"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c","Type":"ContainerStarted","Data":"3080b3d241a272b43edf4e8d072f9a1fe87a6e90b18d355e783284b2cacfed5b"} Nov 03 22:47:59.317597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.317431 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" event={"ID":"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c","Type":"ContainerStarted","Data":"3baf19adf109bd367217e167fa63fdeda380bc64584b63b866437a72798f7c56"} Nov 03 22:47:59.317597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.317460 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:47:59.319343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.319316 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" event={"ID":"206aad65-1e1b-455e-bb78-383be74653af","Type":"ContainerStarted","Data":"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7"} Nov 03 22:47:59.319448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.319357 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" event={"ID":"206aad65-1e1b-455e-bb78-383be74653af","Type":"ContainerStarted","Data":"4572d2d50f5dce92b394769f27f9d4476e36e2fa81cd1477d4ae84a5b3101620"} Nov 03 22:47:59.319448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.319439 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:47:59.437686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.437612 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" podStartSLOduration=2.437594732 podStartE2EDuration="2.437594732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:59.431431663 +0000 UTC m=+1553.168775888" watchObservedRunningTime="2026-11-03 22:47:59.437594732 +0000 UTC m=+1553.174938955" Nov 03 22:47:59.559443 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:47:59.559362 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" podStartSLOduration=2.559342018 podStartE2EDuration="2.559342018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:59.557185507 +0000 UTC m=+1553.294529740" watchObservedRunningTime="2026-11-03 22:47:59.559342018 +0000 UTC m=+1553.296686243" Nov 03 22:48:00.335452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:00.335418 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" event={"ID":"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886","Type":"ContainerStarted","Data":"815d2a4bea7a939f2765f5c220d4916dd00c174a0a7f6514f8bae63eac100cca"} Nov 03 22:48:01.346660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.346622 5051 generic.go:356] "Generic (PLEG): container finished" podID="fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" containerID="815d2a4bea7a939f2765f5c220d4916dd00c174a0a7f6514f8bae63eac100cca" exitCode=0 Nov 03 22:48:01.347214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.346692 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" event={"ID":"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886","Type":"ContainerDied","Data":"815d2a4bea7a939f2765f5c220d4916dd00c174a0a7f6514f8bae63eac100cca"} Nov 03 22:48:01.533377 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.533346 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:48:01.606249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.606195 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\") pod \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " Nov 03 22:48:01.606477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.606343 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\") pod \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " Nov 03 22:48:01.606536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.606487 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\" (UniqueName: \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\") pod \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\" (UID: \"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886\") " Nov 03 22:48:01.606622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.606598 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca" pod "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" (UID: "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:01.607309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.607282 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:01.608873 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.608847 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca" pod "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" (UID: "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:01.609065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.608907 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c" pod "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" (UID: "fc8c4c0e-ee9a-4b87-abd2-7b0483f41886"). InnerVolumeSpecName "kube-api-access-lrr8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:01.709442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.709392 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:01.709442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:01.709441 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr8c\" (UniqueName: \"kubernetes.io/projected/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886-kube-api-access-lrr8c\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:02.366239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:02.366202 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" Nov 03 22:48:02.366239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:02.366232 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks" event={"ID":"fc8c4c0e-ee9a-4b87-abd2-7b0483f41886","Type":"ContainerDied","Data":"71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099"} Nov 03 22:48:02.366697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:02.366319 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71b43b4a8155d94e04b542b91b83e29a48a671a24662ec04a0ab0879730f6099" Nov 03 22:48:08.226537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:08.226492 5051 scope.go:122] "RemoveContainer" containerID="4cd44b8efa944d73a361f4be087e732a619300d5e0e87cc70fd48359390c41c6" Nov 03 22:48:10.339766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:10.339718 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 22:48:10.340427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:10.339817 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 22:48:16.050846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:16.050757 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7"] Nov 03 22:48:16.100031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:16.099966 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e66e53-gf4c7"] Nov 03 22:48:16.412676 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:16.412636 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2978bc99-df8f-41b3-9e03-59f7196c8de3" path="/var/lib/kubelet/pods/2978bc99-df8f-41b3-9e03-59f7196c8de3/volumes" Nov 03 22:48:20.051191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:20.051061 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq"] Nov 03 22:48:20.103649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:20.103602 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1012649d-zndxq"] Nov 03 22:48:20.412442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:20.412398 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5147edd-f9cf-4942-b254-d9544100c207" path="/var/lib/kubelet/pods/b5147edd-f9cf-4942-b254-d9544100c207/volumes" Nov 03 22:48:21.631266 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:48:21.631224 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:48:24.050815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:24.050767 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw"] Nov 03 22:48:24.099951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:24.099899 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a106282-zc9bw"] Nov 03 22:48:24.412405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:24.412369 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec767b9-d2fd-481d-9824-46ab3fb2b290" path="/var/lib/kubelet/pods/5ec767b9-d2fd-481d-9824-46ab3fb2b290/volumes" Nov 03 22:48:30.927979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:30.927922 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:48:30.929715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:30.929688 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" containerName="revoke-keys" Nov 03 22:48:30.929860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:30.929719 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" containerName="revoke-keys" Nov 03 22:48:30.929967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:30.929953 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" containerName="revoke-keys" Nov 03 22:48:30.937257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:30.937203 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.022017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.009516 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ebffae22-openshift-default-dockercfg-hq996\"" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.087562 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109230 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109379 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109486 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109593 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109680 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109814 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnch\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.124051 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.109948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.178076 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.160645 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:48:31.214016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.211815 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.214016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.211924 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252307 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252420 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252489 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnch\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252640 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252760 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.252896 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.253582 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.254298 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.255742 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.258329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.262015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.258906 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.309840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.283276 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.309840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.300972 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.323105 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.322897 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.333063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.329541 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnch\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:31.576922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:31.576782 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:32.071877 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:48:32.071828 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a7cc4e_ed8b_4a83_b7e1_a86f794e5be1.slice/crio-fa1fbcd0b2fd76f65ac6edc821581fb99438b7e4daa07096cc99d3e607e6e059 WatchSource:0}: Error finding container fa1fbcd0b2fd76f65ac6edc821581fb99438b7e4daa07096cc99d3e607e6e059: Status 404 returned error can't find the container with id fa1fbcd0b2fd76f65ac6edc821581fb99438b7e4daa07096cc99d3e607e6e059 Nov 03 22:48:32.075656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.075604 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:32.075796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.075707 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:32.075796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.075755 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:32.076032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.075980 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:48:32.729061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.729018 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerStarted","Data":"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711"} Nov 03 22:48:32.729251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.729082 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerStarted","Data":"fa1fbcd0b2fd76f65ac6edc821581fb99438b7e4daa07096cc99d3e607e6e059"} Nov 03 22:48:32.883199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:32.883134 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podStartSLOduration=2.883115429 podStartE2EDuration="2.883115429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:32.880954408 +0000 UTC m=+1586.618298633" watchObservedRunningTime="2026-11-03 22:48:32.883115429 +0000 UTC m=+1586.620459656" Nov 03 22:48:33.577645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:33.577606 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:33.586339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:33.586306 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:33.739865 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:33.739828 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:33.741280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:33.741251 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:48:36.051169 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:36.051132 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx"] Nov 03 22:48:36.100315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:36.100264 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31659a80-4j5zx"] Nov 03 22:48:36.411731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:36.411694 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbf2e61-1345-4f5e-9710-4bddac89800a" path="/var/lib/kubelet/pods/6fbf2e61-1345-4f5e-9710-4bddac89800a/volumes" Nov 03 22:48:39.285017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.284459 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:48:39.299868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.299640 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.350720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.350677 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:48:39.451101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.450968 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ebffae22\"" Nov 03 22:48:39.452025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.451666 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ebffae22-dockercfg-xhnxx\"" Nov 03 22:48:39.588094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.587845 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.588094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.587948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d459b\" (UniqueName: \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.744086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.741576 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.744086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.739983 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.744086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.741760 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d459b\" (UniqueName: \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.810654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.810534 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d459b\" (UniqueName: \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\") pod \"payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:39.934092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:39.934046 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:40.337963 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:48:40.337926 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba3a7fc_dea8_4d2e_ad36_4dff4db7af12.slice/crio-b2d773ec8425edace6231c7ebb76cd138d27ce5189b78e02b877dd8031810206 WatchSource:0}: Error finding container b2d773ec8425edace6231c7ebb76cd138d27ce5189b78e02b877dd8031810206: Status 404 returned error can't find the container with id b2d773ec8425edace6231c7ebb76cd138d27ce5189b78e02b877dd8031810206 Nov 03 22:48:40.338505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.338459 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:48:40.587023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.586898 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks"] Nov 03 22:48:40.595158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.595111 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cb89500-dw4ks"] Nov 03 22:48:40.913826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.913779 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" event={"ID":"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12","Type":"ContainerStarted","Data":"69d9506e035f6eee2a658fdcab60bc86f2828debb19f7c2f5b301e82954262d5"} Nov 03 22:48:40.914063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.913877 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" event={"ID":"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12","Type":"ContainerStarted","Data":"b2d773ec8425edace6231c7ebb76cd138d27ce5189b78e02b877dd8031810206"} Nov 03 22:48:40.914063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:40.913900 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:42.417863 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:42.417827 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8c4c0e-ee9a-4b87-abd2-7b0483f41886" path="/var/lib/kubelet/pods/fc8c4c0e-ee9a-4b87-abd2-7b0483f41886/volumes" Nov 03 22:48:45.045252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:45.045171 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" podStartSLOduration=6.04515076 podStartE2EDuration="6.04515076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:41.033054216 +0000 UTC m=+1594.770398443" watchObservedRunningTime="2026-11-03 22:48:45.04515076 +0000 UTC m=+1598.782494986" Nov 03 22:48:45.049625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:45.049594 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq"] Nov 03 22:48:45.103143 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:45.103100 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642bc8a9-2dtdq"] Nov 03 22:48:46.411292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:46.411257 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b934a3a-264b-40ec-84fe-eb616a21bba1" path="/var/lib/kubelet/pods/2b934a3a-264b-40ec-84fe-eb616a21bba1/volumes" Nov 03 22:48:51.645507 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:48:51.645463 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:48:51.928289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:51.928200 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:48:53.052390 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:53.052347 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh"] Nov 03 22:48:53.102044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:53.101971 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e4fc87-d2cqh"] Nov 03 22:48:54.422246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.422201 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab74d04d-bafc-49e4-a9c0-cfaa24be4626" path="/var/lib/kubelet/pods/ab74d04d-bafc-49e4-a9c0-cfaa24be4626/volumes" Nov 03 22:48:54.614657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.614609 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 22:48:54.619951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.619919 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657090 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657243 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657351 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcnm\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657393 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657575 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657719 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657751 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657877 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.657957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.657912 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.711096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.710959 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-eb3c33-openshift-default-dockercfg-h4x6n\"" Nov 03 22:48:54.771664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.771618 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.771886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.771678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.771886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.771740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.771886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.771773 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772004 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772062 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772120 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcnm\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772171 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772290 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.772938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.772913 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.775803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.773213 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.776189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.776150 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.777139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.776799 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.777978 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.777936 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.786826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.786696 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 22:48:54.810493 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.810454 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.810721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.810618 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.840516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.840462 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcnm\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.842123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.842077 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\") pod \"e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:54.947409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:54.939428 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:55.478268 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:48:55.478239 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a6af0b_f09e_4b53_be12_24c0a664efbf.slice/crio-c193bc3d4bcf41470b587d6e6860c41456618eb5ac8987e122d3c1740c5e5a02 WatchSource:0}: Error finding container c193bc3d4bcf41470b587d6e6860c41456618eb5ac8987e122d3c1740c5e5a02: Status 404 returned error can't find the container with id c193bc3d4bcf41470b587d6e6860c41456618eb5ac8987e122d3c1740c5e5a02 Nov 03 22:48:55.481535 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:55.481510 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 22:48:55.481795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:55.481760 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:55.481852 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:55.481830 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:55.481891 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:55.481860 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:56.050887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.050841 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d"] Nov 03 22:48:56.057656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.057607 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bd3e364-csr7d"] Nov 03 22:48:56.301914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.301817 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerStarted","Data":"e6a1df71ba436dad6997906d4ff354cc714d77c8950f4fef071ab22f85336432"} Nov 03 22:48:56.301914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.301888 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerStarted","Data":"c193bc3d4bcf41470b587d6e6860c41456618eb5ac8987e122d3c1740c5e5a02"} Nov 03 22:48:56.413322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.413280 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4166410-27db-4474-b701-9a5a4492d5db" path="/var/lib/kubelet/pods/b4166410-27db-4474-b701-9a5a4492d5db/volumes" Nov 03 22:48:56.423941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.423871 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podStartSLOduration=2.423852718 podStartE2EDuration="2.423852718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:56.422108083 +0000 UTC m=+1610.159452310" watchObservedRunningTime="2026-11-03 22:48:56.423852718 +0000 UTC m=+1610.161196954" Nov 03 22:48:56.944356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.944315 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:56.951712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:56.951673 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:57.313102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:57.313005 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:48:57.314234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:48:57.314214 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:49:00.063399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:00.063358 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5"] Nov 03 22:49:00.206872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:00.203960 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2263eb-phzh5"] Nov 03 22:49:00.415521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:00.415483 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e9f5d7-c79e-43ef-83c6-4c6836079db5" path="/var/lib/kubelet/pods/70e9f5d7-c79e-43ef-83c6-4c6836079db5/volumes" Nov 03 22:49:01.160442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:01.160382 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df"] Nov 03 22:49:01.211192 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:01.211072 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv"] Nov 03 22:49:01.222545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:01.222417 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ce2642-k27df"] Nov 03 22:49:01.255922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:01.255877 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f253afa-h9nkv"] Nov 03 22:49:02.411407 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:02.411365 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55446179-8462-49a6-a492-8271e8bb0501" path="/var/lib/kubelet/pods/55446179-8462-49a6-a492-8271e8bb0501/volumes" Nov 03 22:49:02.412333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:02.412306 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d868c385-088e-4a61-a6a4-2e1cdc7a0d6c" path="/var/lib/kubelet/pods/d868c385-088e-4a61-a6a4-2e1cdc7a0d6c/volumes" Nov 03 22:49:06.031671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.031610 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 22:49:06.047124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.043939 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.092090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.092048 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 22:49:06.129349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.129299 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmvw\" (UniqueName: \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.129616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.129564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.132620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.132582 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-eb3c33\"" Nov 03 22:49:06.132824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.132640 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-eb3c33-dockercfg-twkxc\"" Nov 03 22:49:06.268181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.264707 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmvw\" (UniqueName: \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.268181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.264863 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.268181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.267559 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.304696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.304537 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 22:49:06.322813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.322197 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.335551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.335511 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 22:49:06.338749 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.338717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmvw\" (UniqueName: \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\") pod \"payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.372816 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.372654 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.373040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.372974 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblc6\" (UniqueName: \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.373427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.373387 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:06.482071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.475599 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.482071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.476010 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblc6\" (UniqueName: \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.482071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.477282 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.609203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.600239 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblc6\" (UniqueName: \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\") pod \"payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.676340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.676291 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:06.823266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:06.821746 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 22:49:07.079370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.079332 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 22:49:07.082552 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:49:07.082509 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb933528a_d3df_4255_940d_6fc634b58699.slice/crio-af1b6e349bb0d4152485ba934be05cc6f4faea37e43adee8bec22ae4a3acb252 WatchSource:0}: Error finding container af1b6e349bb0d4152485ba934be05cc6f4faea37e43adee8bec22ae4a3acb252: Status 404 returned error can't find the container with id af1b6e349bb0d4152485ba934be05cc6f4faea37e43adee8bec22ae4a3acb252 Nov 03 22:49:07.475799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.475692 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" event={"ID":"b933528a-d3df-4255-940d-6fc634b58699","Type":"ContainerStarted","Data":"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e"} Nov 03 22:49:07.475799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.475762 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" event={"ID":"b933528a-d3df-4255-940d-6fc634b58699","Type":"ContainerStarted","Data":"af1b6e349bb0d4152485ba934be05cc6f4faea37e43adee8bec22ae4a3acb252"} Nov 03 22:49:07.475799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.475792 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:07.478168 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.478135 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" event={"ID":"42a148d1-c86d-49c7-b93b-95ec72de33a1","Type":"ContainerStarted","Data":"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4"} Nov 03 22:49:07.478326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.478184 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" event={"ID":"42a148d1-c86d-49c7-b93b-95ec72de33a1","Type":"ContainerStarted","Data":"4e9c0a3f059d7a996d8864ad7c1fc3794e35e25f4e1d4ce566a432111993c9b1"} Nov 03 22:49:07.478326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.478208 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:07.594059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.593944 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" podStartSLOduration=1.593923889 podStartE2EDuration="1.593923889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:07.59226168 +0000 UTC m=+1621.329605907" watchObservedRunningTime="2026-11-03 22:49:07.593923889 +0000 UTC m=+1621.331268117" Nov 03 22:49:07.711761 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:07.710658 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" podStartSLOduration=2.710638195 podStartE2EDuration="2.710638195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:07.707367822 +0000 UTC m=+1621.444712049" watchObservedRunningTime="2026-11-03 22:49:07.710638195 +0000 UTC m=+1621.447982421" Nov 03 22:49:08.284867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:08.284799 5051 scope.go:122] "RemoveContainer" containerID="2418fca28a464e2194461f67f14486bf3c3ca5f0f53949e6f445bba2077d13ba" Nov 03 22:49:08.301218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:08.301189 5051 scope.go:122] "RemoveContainer" containerID="04f3463ec13b92360f732e2cf072384ba7c5666905e91a08619195b569ef8d4b" Nov 03 22:49:08.320020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:08.319977 5051 scope.go:122] "RemoveContainer" containerID="b10c00e0e00cf50590491ef58240386d73186abd130487b8e6774bac0ca1ff98" Nov 03 22:49:15.054114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.054066 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5"] Nov 03 22:49:15.105856 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.105802 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0282acde-wqnw5"] Nov 03 22:49:15.323325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.323221 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg"] Nov 03 22:49:15.329681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.329650 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.377184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.377134 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.377393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.377375 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7vk\" (UniqueName: \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.377457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.377413 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.393218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.382320 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:49:15.393218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.382464 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:49:15.393218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.391130 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg"] Nov 03 22:49:15.479065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.479013 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7vk\" (UniqueName: \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.479283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.479078 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.479410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.479382 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.480101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.480072 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.482288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.482243 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.539936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.539902 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7vk\" (UniqueName: \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\") pod \"maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:15.659507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:15.659455 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:16.069468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:16.069426 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg"] Nov 03 22:49:16.069949 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:49:16.069415 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41357ea_0d74_4d39_820e_1aab8651edd8.slice/crio-ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e WatchSource:0}: Error finding container ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e: Status 404 returned error can't find the container with id ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e Nov 03 22:49:16.414350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:16.414313 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34642db4-9235-4535-b360-c81d2168b284" path="/var/lib/kubelet/pods/34642db4-9235-4535-b360-c81d2168b284/volumes" Nov 03 22:49:16.577372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:16.577323 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" event={"ID":"b41357ea-0d74-4d39-820e-1aab8651edd8","Type":"ContainerStarted","Data":"3ab6e1e8881670a24d1f620812a43d14f2507014ffc3a4a71636b7e4c2f6bef5"} Nov 03 22:49:16.577606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:16.577392 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" event={"ID":"b41357ea-0d74-4d39-820e-1aab8651edd8","Type":"ContainerStarted","Data":"ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e"} Nov 03 22:49:16.694318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:16.694181 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" podStartSLOduration=1.6941598359999999 podStartE2EDuration="1.694159836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:16.692053665 +0000 UTC m=+1630.429397889" watchObservedRunningTime="2026-11-03 22:49:16.694159836 +0000 UTC m=+1630.431504062" Nov 03 22:49:17.058293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.058200 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj"] Nov 03 22:49:17.067491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.067448 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7"] Nov 03 22:49:17.108654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.108614 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9d704-t8nw7"] Nov 03 22:49:17.114234 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.114189 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20cc7e34-ljgcj"] Nov 03 22:49:17.600655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.600619 5051 generic.go:356] "Generic (PLEG): container finished" podID="b41357ea-0d74-4d39-820e-1aab8651edd8" containerID="3ab6e1e8881670a24d1f620812a43d14f2507014ffc3a4a71636b7e4c2f6bef5" exitCode=0 Nov 03 22:49:17.600898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:17.600714 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" event={"ID":"b41357ea-0d74-4d39-820e-1aab8651edd8","Type":"ContainerDied","Data":"3ab6e1e8881670a24d1f620812a43d14f2507014ffc3a4a71636b7e4c2f6bef5"} Nov 03 22:49:18.054032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.053948 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85"] Nov 03 22:49:18.059771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.059728 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61931a53-ssf85"] Nov 03 22:49:18.413579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.413548 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8380d0c7-517d-48a2-9ad7-e5951a060780" path="/var/lib/kubelet/pods/8380d0c7-517d-48a2-9ad7-e5951a060780/volumes" Nov 03 22:49:18.414183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.414165 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99c0a29-be2a-4799-841d-ea0eaeadbde9" path="/var/lib/kubelet/pods/a99c0a29-be2a-4799-841d-ea0eaeadbde9/volumes" Nov 03 22:49:18.414669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.414656 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4db3dff-af6c-482e-92fe-54c895cece64" path="/var/lib/kubelet/pods/b4db3dff-af6c-482e-92fe-54c895cece64/volumes" Nov 03 22:49:18.493159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.493128 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 22:49:18.493393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.493208 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 22:49:18.856367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.856336 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:18.957131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.956980 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\" (UniqueName: \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\") pod \"b41357ea-0d74-4d39-820e-1aab8651edd8\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " Nov 03 22:49:18.957329 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.957179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\") pod \"b41357ea-0d74-4d39-820e-1aab8651edd8\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " Nov 03 22:49:18.957403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.957356 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\") pod \"b41357ea-0d74-4d39-820e-1aab8651edd8\" (UID: \"b41357ea-0d74-4d39-820e-1aab8651edd8\") " Nov 03 22:49:18.957881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.957787 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca" pod "b41357ea-0d74-4d39-820e-1aab8651edd8" (UID: "b41357ea-0d74-4d39-820e-1aab8651edd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:18.958607 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.958579 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:18.959788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.959756 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk" pod "b41357ea-0d74-4d39-820e-1aab8651edd8" (UID: "b41357ea-0d74-4d39-820e-1aab8651edd8"). InnerVolumeSpecName "kube-api-access-9x7vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:18.959904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:18.959789 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca" pod "b41357ea-0d74-4d39-820e-1aab8651edd8" (UID: "b41357ea-0d74-4d39-820e-1aab8651edd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:19.051019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.050952 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv"] Nov 03 22:49:19.061333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.061288 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41357ea-0d74-4d39-820e-1aab8651edd8-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:19.061333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.061333 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x7vk\" (UniqueName: \"kubernetes.io/projected/b41357ea-0d74-4d39-820e-1aab8651edd8-kube-api-access-9x7vk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:19.075471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.075428 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38c5768e-m7btv"] Nov 03 22:49:19.622908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.622876 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" Nov 03 22:49:19.623478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.622898 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg" event={"ID":"b41357ea-0d74-4d39-820e-1aab8651edd8","Type":"ContainerDied","Data":"ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e"} Nov 03 22:49:19.623478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:19.622978 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3d5065288c77ec446fa3f8851da2a5db458347d0ee3de8249508a4aee8209e" Nov 03 22:49:20.412006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:20.411961 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7eaaa1c-9772-4fda-a09f-1d8e84024a92" path="/var/lib/kubelet/pods/c7eaaa1c-9772-4fda-a09f-1d8e84024a92/volumes" Nov 03 22:49:21.628319 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:49:21.628277 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:49:26.438272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.438218 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd"] Nov 03 22:49:26.439383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.439354 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b41357ea-0d74-4d39-820e-1aab8651edd8" containerName="revoke-keys" Nov 03 22:49:26.439383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.439384 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41357ea-0d74-4d39-820e-1aab8651edd8" containerName="revoke-keys" Nov 03 22:49:26.439617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.439581 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41357ea-0d74-4d39-820e-1aab8651edd8" containerName="revoke-keys" Nov 03 22:49:26.443576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.443528 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.502358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.502319 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd"] Nov 03 22:49:26.543334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.543294 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:49:26.543653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.543518 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:49:26.572226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.572146 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48pk\" (UniqueName: \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.572516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.572265 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.572770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.572742 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.675404 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.675361 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.675619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.675456 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48pk\" (UniqueName: \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.675619 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.675496 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.676317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.676290 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.678363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.678331 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.738396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.738242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48pk\" (UniqueName: \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\") pod \"maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:26.761250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:26.761195 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:27.197942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:27.197898 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd"] Nov 03 22:49:27.198755 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:49:27.198718 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5155843b_11b6_477b_843b_7de1225469af.slice/crio-144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc WatchSource:0}: Error finding container 144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc: Status 404 returned error can't find the container with id 144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc Nov 03 22:49:27.720881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:27.720841 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" event={"ID":"5155843b-11b6-477b-843b-7de1225469af","Type":"ContainerStarted","Data":"2ade9c03dcebb86b1ff90409cafe5911dc7959ede2a2f3cc62686ac36cc93c74"} Nov 03 22:49:27.721440 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:27.720898 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" event={"ID":"5155843b-11b6-477b-843b-7de1225469af","Type":"ContainerStarted","Data":"144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc"} Nov 03 22:49:27.841364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:27.841303 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" podStartSLOduration=1.841284444 podStartE2EDuration="1.841284444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:27.835325959 +0000 UTC m=+1641.572670186" watchObservedRunningTime="2026-11-03 22:49:27.841284444 +0000 UTC m=+1641.578628670" Nov 03 22:49:28.144014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:28.132205 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn"] Nov 03 22:49:28.207653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:28.201110 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d158bf80-tgcvn"] Nov 03 22:49:28.417136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:28.417049 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511cc975-ebbb-47a6-ba87-fd04c4eaacd0" path="/var/lib/kubelet/pods/511cc975-ebbb-47a6-ba87-fd04c4eaacd0/volumes" Nov 03 22:49:28.747486 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:28.747378 5051 generic.go:356] "Generic (PLEG): container finished" podID="5155843b-11b6-477b-843b-7de1225469af" containerID="2ade9c03dcebb86b1ff90409cafe5911dc7959ede2a2f3cc62686ac36cc93c74" exitCode=0 Nov 03 22:49:28.747978 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:28.747576 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" event={"ID":"5155843b-11b6-477b-843b-7de1225469af","Type":"ContainerDied","Data":"2ade9c03dcebb86b1ff90409cafe5911dc7959ede2a2f3cc62686ac36cc93c74"} Nov 03 22:49:29.050671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.050567 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6"] Nov 03 22:49:29.057159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.057113 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9833a00-tgqg6"] Nov 03 22:49:29.962774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.962737 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:29.972807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.972774 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\" (UniqueName: \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\") pod \"5155843b-11b6-477b-843b-7de1225469af\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " Nov 03 22:49:29.973223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.973191 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\") pod \"5155843b-11b6-477b-843b-7de1225469af\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " Nov 03 22:49:29.973363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.973331 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\") pod \"5155843b-11b6-477b-843b-7de1225469af\" (UID: \"5155843b-11b6-477b-843b-7de1225469af\") " Nov 03 22:49:29.973722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.973692 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca" pod "5155843b-11b6-477b-843b-7de1225469af" (UID: "5155843b-11b6-477b-843b-7de1225469af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:29.974589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.974555 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5155843b-11b6-477b-843b-7de1225469af-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:29.975640 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.975610 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca" pod "5155843b-11b6-477b-843b-7de1225469af" (UID: "5155843b-11b6-477b-843b-7de1225469af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:29.975732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:29.975635 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk" pod "5155843b-11b6-477b-843b-7de1225469af" (UID: "5155843b-11b6-477b-843b-7de1225469af"). InnerVolumeSpecName "kube-api-access-w48pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:30.076043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.075911 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5155843b-11b6-477b-843b-7de1225469af-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:30.076043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.075950 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48pk\" (UniqueName: \"kubernetes.io/projected/5155843b-11b6-477b-843b-7de1225469af-kube-api-access-w48pk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:49:30.414391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.414357 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15ffb23-511d-403a-a0bc-a62e1a3662c3" path="/var/lib/kubelet/pods/f15ffb23-511d-403a-a0bc-a62e1a3662c3/volumes" Nov 03 22:49:30.782348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.782257 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" Nov 03 22:49:30.782651 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.782345 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd" event={"ID":"5155843b-11b6-477b-843b-7de1225469af","Type":"ContainerDied","Data":"144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc"} Nov 03 22:49:30.782651 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:30.782422 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="144c189006da1cf3d9557d88afe306782d0cb59eb1559f3beecff61823ac54bc" Nov 03 22:49:31.054620 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:31.054505 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj"] Nov 03 22:49:31.064177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:31.064126 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6321b11-rp4sj"] Nov 03 22:49:32.415274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:32.415236 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa414417-9314-4a0b-b581-78c792f7159a" path="/var/lib/kubelet/pods/fa414417-9314-4a0b-b581-78c792f7159a/volumes" Nov 03 22:49:33.050422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:33.050376 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2"] Nov 03 22:49:33.100827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:33.100778 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64093830-j8nm2"] Nov 03 22:49:34.412757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:34.412721 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90985c4a-5bac-4e38-bf82-fa667dcf5e86" path="/var/lib/kubelet/pods/90985c4a-5bac-4e38-bf82-fa667dcf5e86/volumes" Nov 03 22:49:42.053125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:42.053087 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8"] Nov 03 22:49:42.105301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:42.105254 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34ad0d5-nx5f8"] Nov 03 22:49:42.410649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:42.410617 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7a1e3f-2e64-432d-8a56-d51db575b5cd" path="/var/lib/kubelet/pods/0e7a1e3f-2e64-432d-8a56-d51db575b5cd/volumes" Nov 03 22:49:43.052475 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:43.052429 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl"] Nov 03 22:49:43.102556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:43.102504 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v"] Nov 03 22:49:43.109054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:43.108946 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e5764a-zqhrl"] Nov 03 22:49:43.114537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:43.114494 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33566b61-xkr9v"] Nov 03 22:49:44.411414 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:44.411370 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73544a1d-21b6-4216-8ff2-40b63b12a9bc" path="/var/lib/kubelet/pods/73544a1d-21b6-4216-8ff2-40b63b12a9bc/volumes" Nov 03 22:49:44.412253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:44.412229 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745d7bdc-c22b-4c3f-be8e-66331f5f8320" path="/var/lib/kubelet/pods/745d7bdc-c22b-4c3f-be8e-66331f5f8320/volumes" Nov 03 22:49:51.630450 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:49:51.630345 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:49:54.052186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:54.052134 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn"] Nov 03 22:49:54.101102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:54.101048 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9f62432-cmsdn"] Nov 03 22:49:54.411536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:54.411494 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fce95cc-b0a4-4232-ad93-380c0627a425" path="/var/lib/kubelet/pods/2fce95cc-b0a4-4232-ad93-380c0627a425/volumes" Nov 03 22:49:57.468544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.468498 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 22:49:57.469838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.469806 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5155843b-11b6-477b-843b-7de1225469af" containerName="revoke-keys" Nov 03 22:49:57.469838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.469840 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5155843b-11b6-477b-843b-7de1225469af" containerName="revoke-keys" Nov 03 22:49:57.470116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.470099 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5155843b-11b6-477b-843b-7de1225469af" containerName="revoke-keys" Nov 03 22:49:57.474944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.474914 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.529251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.529202 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4a1480-openshift-default-dockercfg-hgjct\"" Nov 03 22:49:57.593012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.591620 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 22:49:57.613891 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.613831 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.614180 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.613921 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.633537 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.633763 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.634095 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.634172 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ssg\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.634420 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.634488 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.643029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.634664 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736476 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736518 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736604 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736627 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736654 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736682 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.736831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736761 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736851 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.736899 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ssg\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.737063 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.737099 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.737329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737422 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.737404 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.737518 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.737500 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.739794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.739767 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.739980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.739962 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.799942 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.799257 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ssg\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:57.800664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:57.800581 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\") pod \"e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:58.097848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:58.097699 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:49:58.564488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:58.564163 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 22:49:58.568726 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:49:58.568679 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31cf560_8002_451f_bc14_e2f8792597d3.slice/crio-092656163c1cdc0581d8124891c8ac1b824d7e3121901bf570f1bf87269d41f4 WatchSource:0}: Error finding container 092656163c1cdc0581d8124891c8ac1b824d7e3121901bf570f1bf87269d41f4: Status 404 returned error can't find the container with id 092656163c1cdc0581d8124891c8ac1b824d7e3121901bf570f1bf87269d41f4 Nov 03 22:49:58.572719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:58.572662 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:49:58.572845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:58.572778 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:49:58.572845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:58.572829 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:49:59.182481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:59.182437 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerStarted","Data":"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b"} Nov 03 22:49:59.182710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:59.182507 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerStarted","Data":"092656163c1cdc0581d8124891c8ac1b824d7e3121901bf570f1bf87269d41f4"} Nov 03 22:49:59.307936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:49:59.307876 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podStartSLOduration=2.307858469 podStartE2EDuration="2.307858469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:59.30211511 +0000 UTC m=+1673.039459343" watchObservedRunningTime="2026-11-03 22:49:59.307858469 +0000 UTC m=+1673.045202771" Nov 03 22:50:00.098056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:00.098022 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:50:00.104576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:00.104545 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:50:00.193429 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:00.193380 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:50:00.194373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:00.194352 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:50:08.385162 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.385127 5051 scope.go:122] "RemoveContainer" containerID="9b0de1a780d02f3b3ee56d7dee18ac93511288f504564e5aef51714018b1f46d" Nov 03 22:50:08.399001 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.398962 5051 scope.go:122] "RemoveContainer" containerID="1792d4065fb7ea7e8b5b492da22e3de48da13e017d4bb2154b6e4567e938b67d" Nov 03 22:50:08.411489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.411470 5051 scope.go:122] "RemoveContainer" containerID="25ded95454b8bb69a168ba5b8202f2fdbb092b7f827cc130f859a17c0571b9b7" Nov 03 22:50:08.421378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.421351 5051 scope.go:122] "RemoveContainer" containerID="19e4d2e91ad844ec6d8ea3ffbed73b81e67ec4aa7c993534cd0bfd747d99f2ef" Nov 03 22:50:08.431254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.431226 5051 scope.go:122] "RemoveContainer" containerID="482a22410cf4f2037394908b475bbc62514720946b7eed4b0f82aaf583857fd6" Nov 03 22:50:08.440691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.440673 5051 scope.go:122] "RemoveContainer" containerID="c07514cf8e3b6fe7ba54ecaf3f522aaac87009cbf0de0cc75a5ea02168ded3a0" Nov 03 22:50:08.451238 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.451218 5051 scope.go:122] "RemoveContainer" containerID="3e19f69d94f7cbe95b20c6ee90b3755d557775e7fed6262339611cec8849addd" Nov 03 22:50:08.492875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.492745 5051 scope.go:122] "RemoveContainer" containerID="a031b56056df5c01deeb016f497aa1476ead607a245eee4d6bf8bf7d64b2f133" Nov 03 22:50:08.504557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.504527 5051 scope.go:122] "RemoveContainer" containerID="6080f1b16f3aa68f92dea59e6b1815f73cc9ac64f767f2719651fbc982b11b17" Nov 03 22:50:08.515823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:08.515807 5051 scope.go:122] "RemoveContainer" containerID="a6cbdbcf0b3b4fd3a421a89366ad0e739d2afd6748b19a3968b416280f500df3" Nov 03 22:50:10.050634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:10.050601 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj"] Nov 03 22:50:10.099548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:10.099521 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5994de05-7nrnj"] Nov 03 22:50:10.409830 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:10.409800 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350b5aab-ff83-4af5-8d9c-aca817d3cec4" path="/var/lib/kubelet/pods/350b5aab-ff83-4af5-8d9c-aca817d3cec4/volumes" Nov 03 22:50:18.168789 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.168719 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg"] Nov 03 22:50:18.216323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.216296 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26b8d994-s9rfg"] Nov 03 22:50:18.413243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.413207 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41357ea-0d74-4d39-820e-1aab8651edd8" path="/var/lib/kubelet/pods/b41357ea-0d74-4d39-820e-1aab8651edd8/volumes" Nov 03 22:50:18.554599 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.554522 5051 generic.go:356] "Generic (PLEG): container finished" podID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerID="bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711" exitCode=0 Nov 03 22:50:18.554753 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.554590 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerDied","Data":"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711"} Nov 03 22:50:18.555086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.555046 5051 scope.go:122] "RemoveContainer" containerID="bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711" Nov 03 22:50:18.556854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.556823 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:18.556949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.556899 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:18.556949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.556932 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:18.743822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:18.739698 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:50:19.587266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:19.587219 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerStarted","Data":"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d"} Nov 03 22:50:20.046216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.046176 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:50:20.052462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.052440 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.111495 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll"] Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.118455 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ea1be0-openshift-default-dockercfg-jpbnk\"" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.119507 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.119577 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.119693 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.119723 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.119880 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.120137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.120167 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.120200 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5sq\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.131013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.120236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312485 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312538 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312577 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5sq\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312613 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312730 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312875 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313091 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.312903 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.313123 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313714 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.313668 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.313828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.313792 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8f44bd-hcmll"] Nov 03 22:50:20.370134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.319853 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.370134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.320505 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.370134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.335453 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.370134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.336065 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.646746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.639022 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.646746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.639508 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.646746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.640496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5sq\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.649673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.649639 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\") pod \"e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.750485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.750448 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:20.788623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.755669 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:50:20.788623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.755733 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 22:50:20.827878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.827842 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26b6252-36e3-4fd7-8b89-6fd7fb902a10" path="/var/lib/kubelet/pods/a26b6252-36e3-4fd7-8b89-6fd7fb902a10/volumes" Nov 03 22:50:20.828530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.828509 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:50:20.828618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:20.828579 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:50:21.338044 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:50:21.338011 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83820620_a67c_4073_a7e7_3d05224575a8.slice/crio-dc4a0a16a45a77b8a35ef48566f2aad0d4e8a863040c25d61167937c9ef8a2a8 WatchSource:0}: Error finding container dc4a0a16a45a77b8a35ef48566f2aad0d4e8a863040c25d61167937c9ef8a2a8: Status 404 returned error can't find the container with id dc4a0a16a45a77b8a35ef48566f2aad0d4e8a863040c25d61167937c9ef8a2a8 Nov 03 22:50:21.338195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.338173 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:50:21.340907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.340872 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:21.341042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.340941 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:21.341042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.340970 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:21.439291 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.439262 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:50:21.439513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.439492 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" containerID="cri-o://a13ddd8012c8e46d0b7d7d82bb26444a11702498c736db5d590dcee6b1bbbe41" gracePeriod=30 Nov 03 22:50:21.445748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.445721 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:50:21.445949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.445926 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" containerID="cri-o://f3e6cf77a9dbfec6f2254068e7f212136ecc1bf4155363fa0a8da607f3bb9345" gracePeriod=30 Nov 03 22:50:21.452103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.450444 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:50:21.452103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.450641 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" containerID="cri-o://0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf" gracePeriod=30 Nov 03 22:50:21.455346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.455324 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:50:21.455513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.455494 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" containerID="cri-o://18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c" gracePeriod=30 Nov 03 22:50:21.580014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.579337 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:50:21.580014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.579395 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 22:50:21.643616 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:21.643581 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:50:21.850507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.850463 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerStarted","Data":"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557"} Nov 03 22:50:21.851036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:21.850538 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerStarted","Data":"dc4a0a16a45a77b8a35ef48566f2aad0d4e8a863040c25d61167937c9ef8a2a8"} Nov 03 22:50:22.035149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.035053 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:50:22.035330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.035299 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" containerID="cri-o://b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c" gracePeriod=30 Nov 03 22:50:22.040910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.040883 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:50:22.041355 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.041314 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" containerID="cri-o://1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a" gracePeriod=30 Nov 03 22:50:22.046147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.046124 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:50:22.046336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.046316 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" containerID="cri-o://3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3" gracePeriod=30 Nov 03 22:50:22.051775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.051753 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:50:22.051925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.051908 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" containerID="cri-o://1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946" gracePeriod=30 Nov 03 22:50:22.094503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.090390 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podStartSLOduration=3.090371958 podStartE2EDuration="3.090371958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:22.081328827 +0000 UTC m=+1695.818673055" watchObservedRunningTime="2026-11-03 22:50:22.090371958 +0000 UTC m=+1695.827716188" Nov 03 22:50:22.241261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.240757 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:50:22.241261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.241048 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" containerID="cri-o://abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3" gracePeriod=30 Nov 03 22:50:22.268016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.249434 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:50:22.268016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.249712 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" containerID="cri-o://67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24" gracePeriod=30 Nov 03 22:50:22.593017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.587146 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:50:22.593017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.587206 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 22:50:22.758840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.756179 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:22.785016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.779297 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:22.894097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.893355 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:22.900815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:22.900785 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:50:23.345313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:23.345159 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:23.345313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:23.345207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:23.583489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:23.583425 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:50:23.583694 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:23.583527 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 22:50:24.590436 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:24.590396 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 22:50:24.591153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:24.591115 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 22:50:24.952054 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:24.952019 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd"] Nov 03 22:50:24.970018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:24.969653 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9d7aa682-xlzxd"] Nov 03 22:50:25.445747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:25.445705 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:25.446072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:25.445781 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:25.703267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:25.703174 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:50:25.704281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:25.704255 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:50:25.705131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:25.705110 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:50:26.528244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:26.527943 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5155843b-11b6-477b-843b-7de1225469af" path="/var/lib/kubelet/pods/5155843b-11b6-477b-843b-7de1225469af/volumes" Nov 03 22:50:26.875635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:26.873478 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:50:26.941850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:26.941828 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:50:26.971964 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:26.971933 5051 generic.go:356] "Generic (PLEG): container finished" podID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerID="a13ddd8012c8e46d0b7d7d82bb26444a11702498c736db5d590dcee6b1bbbe41" exitCode=0 Nov 03 22:50:26.972316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:26.972296 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" event={"ID":"7a3aa3a7-defd-4022-9551-abf2611d246c","Type":"ContainerDied","Data":"a13ddd8012c8e46d0b7d7d82bb26444a11702498c736db5d590dcee6b1bbbe41"} Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021240 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021314 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021413 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021651 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021685 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021712 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021794 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.023327 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.021915 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\") pod \"642079d7-6d23-43e7-8625-32da34f6ca3a\" (UID: \"642079d7-6d23-43e7-8625-32da34f6ca3a\") " Nov 03 22:50:27.030631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.029198 5051 generic.go:356] "Generic (PLEG): container finished" podID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerID="18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c" exitCode=0 Nov 03 22:50:27.030631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.029353 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" event={"ID":"642079d7-6d23-43e7-8625-32da34f6ca3a","Type":"ContainerDied","Data":"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c"} Nov 03 22:50:27.030631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.029404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" event={"ID":"642079d7-6d23-43e7-8625-32da34f6ca3a","Type":"ContainerDied","Data":"11a2586210836c565849ba29bfd5054aabd5b23d3f525782aaea68c687939d90"} Nov 03 22:50:27.030631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.029437 5051 scope.go:122] "RemoveContainer" containerID="18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c" Nov 03 22:50:27.030631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.029662 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr" Nov 03 22:50:27.033833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.032971 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.033833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.033528 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.033833 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.033766 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.037481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.034202 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.037481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.034218 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.037481 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.034335 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.038771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.038144 5051 generic.go:356] "Generic (PLEG): container finished" podID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerID="f3e6cf77a9dbfec6f2254068e7f212136ecc1bf4155363fa0a8da607f3bb9345" exitCode=0 Nov 03 22:50:27.038771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.038356 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" event={"ID":"5eff83a7-5ef9-4f6c-9949-2202d18946e3","Type":"ContainerDied","Data":"f3e6cf77a9dbfec6f2254068e7f212136ecc1bf4155363fa0a8da607f3bb9345"} Nov 03 22:50:27.042294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.042214 5051 generic.go:356] "Generic (PLEG): container finished" podID="c8669183-3138-4255-b575-24abbd91ba84" containerID="0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf" exitCode=0 Nov 03 22:50:27.043446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.043391 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" Nov 03 22:50:27.043649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.043525 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" event={"ID":"c8669183-3138-4255-b575-24abbd91ba84","Type":"ContainerDied","Data":"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf"} Nov 03 22:50:27.043649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.043617 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl" event={"ID":"c8669183-3138-4255-b575-24abbd91ba84","Type":"ContainerDied","Data":"f1896091c07be613ebbedcdd07945e3a71590295d022662edc084ba013a4f82e"} Nov 03 22:50:27.082917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.067251 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:50:27.082917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.081367 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.082917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.081563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "kube-api-access-24grq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132668 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132793 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132829 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132927 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.132981 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.133137 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.133262 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.133395 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\") pod \"c8669183-3138-4255-b575-24abbd91ba84\" (UID: \"c8669183-3138-4255-b575-24abbd91ba84\") " Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134316 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134338 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134348 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134362 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24grq\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-kube-api-access-24grq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134376 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642079d7-6d23-43e7-8625-32da34f6ca3a-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134385 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134393 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.134945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.134403 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.143287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.143251 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.144200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.144125 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.152038 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.151860 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.152149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.152119 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.152362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.152289 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.154506 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.154396 5051 scope.go:122] "RemoveContainer" containerID="18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c" Nov 03 22:50:27.159544 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:27.159196 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c\": container with ID starting with 18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c not found: ID does not exist" containerID="18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c" Nov 03 22:50:27.159544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.159236 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c"} err="failed to get container status \"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c\": rpc error: code = NotFound desc = could not find container \"18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c\": container with ID starting with 18b8c19f224f5d50f5c49be27d7bb6226c02ad737e44953342195a7d4a0d2f4c not found: ID does not exist" Nov 03 22:50:27.159544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.159260 5051 scope.go:122] "RemoveContainer" containerID="0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf" Nov 03 22:50:27.162381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.161931 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo" pod "642079d7-6d23-43e7-8625-32da34f6ca3a" (UID: "642079d7-6d23-43e7-8625-32da34f6ca3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.162381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.162086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "kube-api-access-6cr8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.167543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.167064 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.170066 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.169408 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.190044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.189979 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token" pod "c8669183-3138-4255-b575-24abbd91ba84" (UID: "c8669183-3138-4255-b575-24abbd91ba84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.235920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.235894 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236025 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236094 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236125 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236176 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236324 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236380 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236439 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.236575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236454 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.236575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236521 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.236679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236612 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236665 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\") pod \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\" (UID: \"5eff83a7-5ef9-4f6c-9949-2202d18946e3\") " Nov 03 22:50:27.236793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236662 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.236887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.236845 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.237805 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.237923 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.237948 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.237964 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.237979 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642079d7-6d23-43e7-8625-32da34f6ca3a-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238040 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238055 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238071 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8669183-3138-4255-b575-24abbd91ba84-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238084 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238098 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr8l\" (UniqueName: \"kubernetes.io/projected/c8669183-3138-4255-b575-24abbd91ba84-kube-api-access-6cr8l\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238113 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8669183-3138-4255-b575-24abbd91ba84-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238193 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8669183-3138-4255-b575-24abbd91ba84-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238208 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238223 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238258 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.238925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.238902 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.243069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.239468 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "kube-api-access-vpqsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.243069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.242974 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.244850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.244823 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy" pod "5eff83a7-5ef9-4f6c-9949-2202d18946e3" (UID: "5eff83a7-5ef9-4f6c-9949-2202d18946e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.340665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.340559 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.340665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.340590 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.340665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.340606 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpqsm\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-kube-api-access-vpqsm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.340665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.340620 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eff83a7-5ef9-4f6c-9949-2202d18946e3-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.449956 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.449849 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:50:27.476536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.476450 5051 scope.go:122] "RemoveContainer" containerID="0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf" Nov 03 22:50:27.494587 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:27.494546 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf\": container with ID starting with 0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf not found: ID does not exist" containerID="0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf" Nov 03 22:50:27.494732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.494594 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf"} err="failed to get container status \"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf\": rpc error: code = NotFound desc = could not find container \"0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf\": container with ID starting with 0ece8bd38f292e03b628b7284a176481602afcebf2c2f08b698f36017769fdbf not found: ID does not exist" Nov 03 22:50:27.543758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.543446 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 22:50:27.543758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.543511 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 22:50:27.549307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549272 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549449 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549351 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549470 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549509 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549578 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549610 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549658 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549815 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549798 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.549874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.549853 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\") pod \"7a3aa3a7-defd-4022-9551-abf2611d246c\" (UID: \"7a3aa3a7-defd-4022-9551-abf2611d246c\") " Nov 03 22:50:27.551476 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.551447 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.551799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.551773 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.554003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.552296 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:50:27.554003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.553578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.554003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.553644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.554003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.553956 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.561330 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.561073 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-2jsvl"] Nov 03 22:50:27.561632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.561599 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "kube-api-access-dp5p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.564216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.564157 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.564302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.564241 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.564534 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.564516 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token" pod "7a3aa3a7-defd-4022-9551-abf2611d246c" (UID: "7a3aa3a7-defd-4022-9551-abf2611d246c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653499 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5p7\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-kube-api-access-dp5p7\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653531 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653546 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653560 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653576 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653589 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653602 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3aa3a7-defd-4022-9551-abf2611d246c-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653615 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3aa3a7-defd-4022-9551-abf2611d246c-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.653652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.653628 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3aa3a7-defd-4022-9551-abf2611d246c-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.691437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.691365 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:50:27.724268 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.723912 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-xvdvr"] Nov 03 22:50:27.748182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.748089 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:50:27.752305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.752277 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:50:27.839131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.839098 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:50:27.850438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.850323 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:50:27.857221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.857193 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.857480 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.857450 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.857647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.857633 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.859454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.859431 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.859665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.859651 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.859769 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.859756 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.859886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.859874 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.860028 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860015 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.860200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860186 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.860339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860327 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.860446 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860435 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.860601 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860589 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.860711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860699 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.860859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860847 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.860956 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860934 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.861106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.860944 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.861228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.861214 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.861430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.861417 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\") pod \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\" (UID: \"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92\") " Nov 03 22:50:27.861628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.861615 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\") pod \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\" (UID: \"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28\") " Nov 03 22:50:27.862600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.862580 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.863373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.863351 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.863656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.863639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.866096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.865515 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.866679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.866232 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.866679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.866433 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.869470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.869274 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.869470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.869341 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.869774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.869672 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.870239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.869972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.871378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.871350 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.871580 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.871562 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.872800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.872773 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.874095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.873517 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "kube-api-access-lk7qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.881902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.881822 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:50:27.885354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.885287 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.885693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.885666 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.886298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.886278 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm" pod "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" (UID: "26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92"). InnerVolumeSpecName "kube-api-access-qlmzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.887838 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.887814 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo" pod "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" (UID: "ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.904170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.904151 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:50:27.963292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.963556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963532 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.963704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963688 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.963813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963570 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.963872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963785 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.963972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.963955 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.964155 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.964138 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.964336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.964319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.964466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.964448 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.964574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.964560 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.973358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.973339 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.973795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.973776 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.974063 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.973984 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.975131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.964489 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.966118 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.966551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.971044 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "kube-api-access-82x77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.971299 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.973644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975087 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.975232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975228 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.975634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975551 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.975634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975612 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.975697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975685 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.975749 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975730 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.975799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975787 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.975857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975842 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.975902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975878 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.975943 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975929 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.975976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.975964 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.976932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.976504 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.976932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.976785 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:27.978703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.978676 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.979284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.979265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.979413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.979399 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.979530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.979517 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.979696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.979681 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\") pod \"f393dcee-c79b-4d3d-9cf5-906209391d94\" (UID: \"f393dcee-c79b-4d3d-9cf5-906209391d94\") " Nov 03 22:50:27.979811 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.979798 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\") pod \"414bff10-4e37-4ef8-be4c-11859f72ae10\" (UID: \"414bff10-4e37-4ef8-be4c-11859f72ae10\") " Nov 03 22:50:27.980207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.980189 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\") pod \"107e0bc0-43c5-40d0-b73a-b34531285f5d\" (UID: \"107e0bc0-43c5-40d0-b73a-b34531285f5d\") " Nov 03 22:50:27.981293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981263 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981421 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981529 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981517 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981607 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981597 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981682 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981757 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414bff10-4e37-4ef8-be4c-11859f72ae10-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981836 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.981927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.981917 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982084 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmzm\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-kube-api-access-qlmzm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982185 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7qm\" (UniqueName: \"kubernetes.io/projected/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-kube-api-access-lk7qm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982278 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982483 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982471 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982572 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982561 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982636 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982719 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982795 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.982937 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.982926 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983065 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983054 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983146 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983283 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983271 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983355 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcee-c79b-4d3d-9cf5-906209391d94-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983434 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983530 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983669 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983757 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82x77\" (UniqueName: \"kubernetes.io/projected/414bff10-4e37-4ef8-be4c-11859f72ae10-kube-api-access-82x77\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983840 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107e0bc0-43c5-40d0-b73a-b34531285f5d-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.983927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.983918 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.984029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.984018 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.984114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.984104 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:27.984361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.984346 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.988020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.985514 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.988020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.985779 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.988020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.985889 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.988020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.987865 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.988284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.988262 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.988368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.984601 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.988639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.988612 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.989923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.989902 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.990313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.990098 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.990313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.990156 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.990313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.990206 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy" pod "414bff10-4e37-4ef8-be4c-11859f72ae10" (UID: "414bff10-4e37-4ef8-be4c-11859f72ae10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:27.990594 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.990555 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:27.991809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.991779 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq" pod "f393dcee-c79b-4d3d-9cf5-906209391d94" (UID: "f393dcee-c79b-4d3d-9cf5-906209391d94"). InnerVolumeSpecName "kube-api-access-s6vxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:27.992473 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:27.992444 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh" pod "107e0bc0-43c5-40d0-b73a-b34531285f5d" (UID: "107e0bc0-43c5-40d0-b73a-b34531285f5d"). InnerVolumeSpecName "kube-api-access-v2bxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:28.056982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.056944 5051 generic.go:356] "Generic (PLEG): container finished" podID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerID="1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a" exitCode=0 Nov 03 22:50:28.057170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.057036 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" Nov 03 22:50:28.057170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.057118 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" event={"ID":"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28","Type":"ContainerDied","Data":"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a"} Nov 03 22:50:28.057170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.057168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz" event={"ID":"ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28","Type":"ContainerDied","Data":"1e17ece8aafe066d39b2a746f9c1a7da9a39c1335fc97a76e445a92f60a80c6e"} Nov 03 22:50:28.057366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.057196 5051 scope.go:122] "RemoveContainer" containerID="1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a" Nov 03 22:50:28.063256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.061758 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" event={"ID":"5eff83a7-5ef9-4f6c-9949-2202d18946e3","Type":"ContainerDied","Data":"d3b65f70c68fba2d7e312551a6f8d6da0bf2d78fc342ed555cca3049f57015b1"} Nov 03 22:50:28.063256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.061871 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr" Nov 03 22:50:28.074057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.067677 5051 generic.go:356] "Generic (PLEG): container finished" podID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerID="abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3" exitCode=0 Nov 03 22:50:28.074057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.067810 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" Nov 03 22:50:28.074057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.067828 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" event={"ID":"107e0bc0-43c5-40d0-b73a-b34531285f5d","Type":"ContainerDied","Data":"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3"} Nov 03 22:50:28.074057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.067899 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq" event={"ID":"107e0bc0-43c5-40d0-b73a-b34531285f5d","Type":"ContainerDied","Data":"8c015327a8f07fe77e08bb0e8907d6b1b9b2ecd5f06df21a935ebaf93fba2c45"} Nov 03 22:50:28.077177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.076936 5051 generic.go:356] "Generic (PLEG): container finished" podID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerID="1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946" exitCode=0 Nov 03 22:50:28.077177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.077072 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" event={"ID":"414bff10-4e37-4ef8-be4c-11859f72ae10","Type":"ContainerDied","Data":"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946"} Nov 03 22:50:28.077177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.077128 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" event={"ID":"414bff10-4e37-4ef8-be4c-11859f72ae10","Type":"ContainerDied","Data":"0936044e150cca46b0f63cce8ac94c279918d871a5ac404a8355fd555d319608"} Nov 03 22:50:28.077354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.077220 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr" Nov 03 22:50:28.091013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.086903 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.091013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.086953 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.091673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.091569 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:28.092174 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.092035 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:28.096380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.092764 5051 generic.go:356] "Generic (PLEG): container finished" podID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerID="67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24" exitCode=0 Nov 03 22:50:28.096380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.092919 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" event={"ID":"f393dcee-c79b-4d3d-9cf5-906209391d94","Type":"ContainerDied","Data":"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24"} Nov 03 22:50:28.096380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.092968 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" event={"ID":"f393dcee-c79b-4d3d-9cf5-906209391d94","Type":"ContainerDied","Data":"a52e690c1c9b70cb46da1693a132eef7cc9821dd45690a5cba522d4fd74cb83b"} Nov 03 22:50:28.096380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.093173 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097116 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097185 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097226 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097301 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097399 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097444 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097569 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\") pod \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\" (UID: \"fa83d4ea-bb36-4dd0-872b-a877e824f2b2\") " Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.097910 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098412 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098468 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098495 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098510 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vxq\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-kube-api-access-s6vxq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098528 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098588 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098602 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098627 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107e0bc0-43c5-40d0-b73a-b34531285f5d-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098641 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098653 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098667 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107e0bc0-43c5-40d0-b73a-b34531285f5d-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098671 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098679 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098692 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098707 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098719 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098731 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098753 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414bff10-4e37-4ef8-be4c-11859f72ae10-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098768 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bxh\" (UniqueName: \"kubernetes.io/projected/107e0bc0-43c5-40d0-b73a-b34531285f5d-kube-api-access-v2bxh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.100044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.098781 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcee-c79b-4d3d-9cf5-906209391d94-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.103120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.103095 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "kube-api-access-tjrvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:28.112686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.107699 5051 generic.go:356] "Generic (PLEG): container finished" podID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerID="b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c" exitCode=0 Nov 03 22:50:28.112686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.107812 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerDied","Data":"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c"} Nov 03 22:50:28.112686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.107861 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" event={"ID":"26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92","Type":"ContainerDied","Data":"81c18db6b99f3b04a107e20e2a31aa3e9b3a848e981cf574573ce3fb7ce52073"} Nov 03 22:50:28.112686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.107939 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2" Nov 03 22:50:28.112686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.109132 5051 scope.go:122] "RemoveContainer" containerID="1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a" Nov 03 22:50:28.113200 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.113174 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a\": container with ID starting with 1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a not found: ID does not exist" containerID="1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a" Nov 03 22:50:28.113200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.113206 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a"} err="failed to get container status \"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a\": rpc error: code = NotFound desc = could not find container \"1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a\": container with ID starting with 1c7811922cd5352ebe981ba4b5831c9d7ac2ad81dfe8f620ba02c6d35024207a not found: ID does not exist" Nov 03 22:50:28.113200 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.113221 5051 scope.go:122] "RemoveContainer" containerID="f3e6cf77a9dbfec6f2254068e7f212136ecc1bf4155363fa0a8da607f3bb9345" Nov 03 22:50:28.115130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.115104 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:28.115316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.115298 5051 generic.go:356] "Generic (PLEG): container finished" podID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerID="3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3" exitCode=0 Nov 03 22:50:28.115467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.115429 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" Nov 03 22:50:28.115554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.115481 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" event={"ID":"fa83d4ea-bb36-4dd0-872b-a877e824f2b2","Type":"ContainerDied","Data":"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3"} Nov 03 22:50:28.115554 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.115525 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4" event={"ID":"fa83d4ea-bb36-4dd0-872b-a877e824f2b2","Type":"ContainerDied","Data":"a35616ffd145b127643343bf5f6d1e21ef5d000f2fb6c5c08129dc92bd3add13"} Nov 03 22:50:28.127094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.126977 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" event={"ID":"7a3aa3a7-defd-4022-9551-abf2611d246c","Type":"ContainerDied","Data":"e3666408f8033363cabdc5bf0416c8167413b98b309ed726abfe327e7d8fdfb3"} Nov 03 22:50:28.127254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.127103 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb" Nov 03 22:50:28.130489 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.130459 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:28.143277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.143259 5051 scope.go:122] "RemoveContainer" containerID="abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3" Nov 03 22:50:28.146399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.146374 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token" pod "fa83d4ea-bb36-4dd0-872b-a877e824f2b2" (UID: "fa83d4ea-bb36-4dd0-872b-a877e824f2b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:28.159369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.159347 5051 scope.go:122] "RemoveContainer" containerID="abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3" Nov 03 22:50:28.159850 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.159823 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3\": container with ID starting with abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3 not found: ID does not exist" containerID="abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3" Nov 03 22:50:28.159973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.159868 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3"} err="failed to get container status \"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3\": rpc error: code = NotFound desc = could not find container \"abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3\": container with ID starting with abb84be36550c974cc8ca3c009906ae0f8c1897a37d4c42ee5f0aabf3690d5d3 not found: ID does not exist" Nov 03 22:50:28.159973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.159893 5051 scope.go:122] "RemoveContainer" containerID="1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946" Nov 03 22:50:28.178622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.178600 5051 scope.go:122] "RemoveContainer" containerID="1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946" Nov 03 22:50:28.178898 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.178869 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946\": container with ID starting with 1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946 not found: ID does not exist" containerID="1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946" Nov 03 22:50:28.179097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.178897 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946"} err="failed to get container status \"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946\": rpc error: code = NotFound desc = could not find container \"1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946\": container with ID starting with 1b79a5cf0047504d58254a3a786c2d7d8f5e8943e49a8982bbce5c980bca0946 not found: ID does not exist" Nov 03 22:50:28.179097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.178916 5051 scope.go:122] "RemoveContainer" containerID="67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202492 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrvp\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-kube-api-access-tjrvp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202524 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202539 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202551 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202564 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.214315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.202575 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa83d4ea-bb36-4dd0-872b-a877e824f2b2-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:28.218673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.218557 5051 scope.go:122] "RemoveContainer" containerID="67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24" Nov 03 22:50:28.220734 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.220704 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24\": container with ID starting with 67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24 not found: ID does not exist" containerID="67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24" Nov 03 22:50:28.220859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.220744 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24"} err="failed to get container status \"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24\": rpc error: code = NotFound desc = could not find container \"67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24\": container with ID starting with 67f76675cf550887c74120e4a683a74628569d56733245f656d2bcbcbfb57d24 not found: ID does not exist" Nov 03 22:50:28.220859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.220769 5051 scope.go:122] "RemoveContainer" containerID="b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c" Nov 03 22:50:28.237358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.237335 5051 scope.go:122] "RemoveContainer" containerID="eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325" Nov 03 22:50:28.248363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.248343 5051 scope.go:122] "RemoveContainer" containerID="b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c" Nov 03 22:50:28.248672 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.248637 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c\": container with ID starting with b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c not found: ID does not exist" containerID="b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c" Nov 03 22:50:28.248793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.248681 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c"} err="failed to get container status \"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c\": rpc error: code = NotFound desc = could not find container \"b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c\": container with ID starting with b887f165b4f8ed94cbf66616eab5ed8b5b554592e215700cb3387ba546fe4a6c not found: ID does not exist" Nov 03 22:50:28.248793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.248708 5051 scope.go:122] "RemoveContainer" containerID="eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325" Nov 03 22:50:28.249131 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.249104 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325\": container with ID starting with eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325 not found: ID does not exist" containerID="eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325" Nov 03 22:50:28.249240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.249139 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325"} err="failed to get container status \"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325\": rpc error: code = NotFound desc = could not find container \"eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325\": container with ID starting with eb431c580954a6216c9b85ef77ea0b59eb5e48df1f47953df4e9e824b6bfa325 not found: ID does not exist" Nov 03 22:50:28.249240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.249159 5051 scope.go:122] "RemoveContainer" containerID="3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3" Nov 03 22:50:28.266272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.266248 5051 scope.go:122] "RemoveContainer" containerID="3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3" Nov 03 22:50:28.273368 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:28.273341 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3\": container with ID starting with 3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3 not found: ID does not exist" containerID="3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3" Nov 03 22:50:28.273540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.273515 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3"} err="failed to get container status \"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3\": rpc error: code = NotFound desc = could not find container \"3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3\": container with ID starting with 3173027204be9b5d473104cfc69106cb6e4f443e6f632f5fb766dda34a3a04e3 not found: ID does not exist" Nov 03 22:50:28.273632 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.273621 5051 scope.go:122] "RemoveContainer" containerID="a13ddd8012c8e46d0b7d7d82bb26444a11702498c736db5d590dcee6b1bbbe41" Nov 03 22:50:28.330716 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.330629 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:50:28.335331 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.335307 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-gjgqz"] Nov 03 22:50:28.410006 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.409968 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" path="/var/lib/kubelet/pods/642079d7-6d23-43e7-8625-32da34f6ca3a/volumes" Nov 03 22:50:28.410532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.410517 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8669183-3138-4255-b575-24abbd91ba84" path="/var/lib/kubelet/pods/c8669183-3138-4255-b575-24abbd91ba84/volumes" Nov 03 22:50:28.410951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.410938 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" path="/var/lib/kubelet/pods/ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28/volumes" Nov 03 22:50:28.864282 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.864251 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:50:28.871275 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.871253 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-p4dg2"] Nov 03 22:50:28.975932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.975888 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:50:28.980542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:28.980518 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-7qx7j"] Nov 03 22:50:29.087111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.087071 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:50:29.094829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.094798 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-8mkq4"] Nov 03 22:50:29.205847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.205757 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:50:29.209944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.209916 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-44frr"] Nov 03 22:50:29.316227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.316189 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:50:29.320191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.320169 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-9v2fq"] Nov 03 22:50:29.434744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.434716 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:50:29.439363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.439345 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-cr5jb"] Nov 03 22:50:29.548270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.548183 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:50:29.552248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:29.552227 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-7bggr"] Nov 03 22:50:30.427507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.427394 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" path="/var/lib/kubelet/pods/107e0bc0-43c5-40d0-b73a-b34531285f5d/volumes" Nov 03 22:50:30.430264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.429175 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" path="/var/lib/kubelet/pods/26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92/volumes" Nov 03 22:50:30.430264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.429697 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" path="/var/lib/kubelet/pods/414bff10-4e37-4ef8-be4c-11859f72ae10/volumes" Nov 03 22:50:30.430637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.430614 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" path="/var/lib/kubelet/pods/5eff83a7-5ef9-4f6c-9949-2202d18946e3/volumes" Nov 03 22:50:30.431292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.431274 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" path="/var/lib/kubelet/pods/7a3aa3a7-defd-4022-9551-abf2611d246c/volumes" Nov 03 22:50:30.432172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.432151 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" path="/var/lib/kubelet/pods/f393dcee-c79b-4d3d-9cf5-906209391d94/volumes" Nov 03 22:50:30.433263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.433242 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" path="/var/lib/kubelet/pods/fa83d4ea-bb36-4dd0-872b-a877e824f2b2/volumes" Nov 03 22:50:30.471333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.471287 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:50:30.472847 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.472819 5051 cpu_manager.go:414] "containerMap: removing container" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.473099 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473077 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.473268 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473251 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" Nov 03 22:50:30.473398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473380 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" Nov 03 22:50:30.473541 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473525 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" Nov 03 22:50:30.473668 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473648 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" Nov 03 22:50:30.473797 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473784 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" Nov 03 22:50:30.473882 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473870 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" Nov 03 22:50:30.473970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.473959 5051 cpu_manager.go:414] "containerMap: removing container" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" Nov 03 22:50:30.474100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474087 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" Nov 03 22:50:30.474222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474204 5051 cpu_manager.go:414] "containerMap: removing container" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" Nov 03 22:50:30.474311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474298 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" Nov 03 22:50:30.474413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474404 5051 cpu_manager.go:414] "containerMap: removing container" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" Nov 03 22:50:30.474574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474556 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" Nov 03 22:50:30.474662 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474652 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" Nov 03 22:50:30.474756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474742 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" Nov 03 22:50:30.474834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474824 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" Nov 03 22:50:30.474901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474891 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" Nov 03 22:50:30.475009 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.474973 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" Nov 03 22:50:30.475123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475110 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" Nov 03 22:50:30.475472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475453 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f393dcee-c79b-4d3d-9cf5-906209391d94" containerName="istio-proxy" Nov 03 22:50:30.475588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475577 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.475671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475663 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.475731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475724 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3c4c30-bf0f-4db7-8024-d7cbf0e90f28" containerName="istio-proxy" Nov 03 22:50:30.475791 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475783 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa83d4ea-bb36-4dd0-872b-a877e824f2b2" containerName="istio-proxy" Nov 03 22:50:30.475862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475854 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8669183-3138-4255-b575-24abbd91ba84" containerName="istio-proxy" Nov 03 22:50:30.475931 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.475922 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="107e0bc0-43c5-40d0-b73a-b34531285f5d" containerName="istio-proxy" Nov 03 22:50:30.476035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.476023 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="642079d7-6d23-43e7-8625-32da34f6ca3a" containerName="istio-proxy" Nov 03 22:50:30.476137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.476120 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="414bff10-4e37-4ef8-be4c-11859f72ae10" containerName="istio-proxy" Nov 03 22:50:30.476213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.476204 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eff83a7-5ef9-4f6c-9949-2202d18946e3" containerName="istio-proxy" Nov 03 22:50:30.476285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.476277 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3aa3a7-defd-4022-9551-abf2611d246c" containerName="istio-proxy" Nov 03 22:50:30.486339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.486319 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.539092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.539061 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ea1be0\"" Nov 03 22:50:30.540078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.540057 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ea1be0-dockercfg-9fdsv\"" Nov 03 22:50:30.578555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.577305 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.578555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.577422 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhw5\" (UniqueName: \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.594018 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.593964 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:50:30.641658 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.641622 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:50:30.642430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.642406 5051 cpu_manager.go:414] "containerMap: removing container" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.642430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.642429 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a5d2fb-5ff7-43f9-a1a5-76de45fc1b92" containerName="istio-proxy" Nov 03 22:50:30.646788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.646763 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.679115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.678983 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.679416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.679392 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhw5\" (UniqueName: \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.679969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.679940 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.718010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.715672 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:50:30.749759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.749689 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhw5\" (UniqueName: \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\") pod \"payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.782423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.782372 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.782605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.782475 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7mx\" (UniqueName: \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.910400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.910358 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.913482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.913443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7mx\" (UniqueName: \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.916615 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.916593 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:30.923124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.920574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:30.984008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:30.983599 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7mx\" (UniqueName: \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\") pod \"payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:31.267084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:31.265963 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:31.391636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:31.391606 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:50:31.673962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:31.673933 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:50:31.674370 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:50:31.674303 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b33f9c7_1553_4fad_8ea0_863239729460.slice/crio-9d742f6917130b9cb715f0b85c6a61e3fa4c38c3b2de038ebec13b9774e48205 WatchSource:0}: Error finding container 9d742f6917130b9cb715f0b85c6a61e3fa4c38c3b2de038ebec13b9774e48205: Status 404 returned error can't find the container with id 9d742f6917130b9cb715f0b85c6a61e3fa4c38c3b2de038ebec13b9774e48205 Nov 03 22:50:32.201829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.201791 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" event={"ID":"3b33f9c7-1553-4fad-8ea0-863239729460","Type":"ContainerStarted","Data":"76aa92bb5aa3a98ae0fc74b612a2f827d48483fdb7594800bd5104c319573cc7"} Nov 03 22:50:32.202053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.201868 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" event={"ID":"3b33f9c7-1553-4fad-8ea0-863239729460","Type":"ContainerStarted","Data":"9d742f6917130b9cb715f0b85c6a61e3fa4c38c3b2de038ebec13b9774e48205"} Nov 03 22:50:32.202153 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.202129 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:32.204490 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.204465 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" event={"ID":"53254326-f0dc-42bf-8edc-e1d7ccb656ef","Type":"ContainerStarted","Data":"246965c6eb4cb3649970b01d22798e4bdbf19f99a769c019c4c8879d681c9e02"} Nov 03 22:50:32.204621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.204506 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" event={"ID":"53254326-f0dc-42bf-8edc-e1d7ccb656ef","Type":"ContainerStarted","Data":"f4ab43aa3fde6ddb6c0db6e41a25c3b6a89bb5a9c26b39e3d226ef7e1cdf5baf"} Nov 03 22:50:32.204621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.204613 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:32.321484 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:32.321426 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" podStartSLOduration=2.321412799 podStartE2EDuration="2.321412799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:32.318122654 +0000 UTC m=+1706.055466870" watchObservedRunningTime="2026-11-03 22:50:32.321412799 +0000 UTC m=+1706.058757024" Nov 03 22:50:35.395525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.395464 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" podStartSLOduration=5.395449326 podStartE2EDuration="5.395449326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:32.45128952 +0000 UTC m=+1706.188633747" watchObservedRunningTime="2026-11-03 22:50:35.395449326 +0000 UTC m=+1709.132793606" Nov 03 22:50:35.397083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.397060 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:50:35.403865 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.403833 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.472846 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.462777 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:50:35.505197 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.505151 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.505659 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.505634 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlc8\" (UniqueName: \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.508795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.507402 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-de5715\"" Nov 03 22:50:35.508795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.507570 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-de5715-dockercfg-6t5s7\"" Nov 03 22:50:35.593021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.589686 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:50:35.610043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.605891 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.610043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.607371 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlc8\" (UniqueName: \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.610043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.607432 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.610043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.608540 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.655102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.653845 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:50:35.687871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.687841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlc8\" (UniqueName: \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\") pod \"payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.709689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.709643 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.709875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.709725 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkl9\" (UniqueName: \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.727025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.723285 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:35.817295 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.817264 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkl9\" (UniqueName: \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.818509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.818483 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.819363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.819347 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.919084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.908677 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkl9\" (UniqueName: \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\") pod \"payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:35.979923 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:35.979888 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:36.163102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:36.158969 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:50:36.616578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:36.616534 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:50:36.616953 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:50:36.616625 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd44189a_405c_4974_b08b_e71ebb0ceb20.slice/crio-18ae576d98fdf0d4f113fb801cd36c378ce189d1f4c6793c3548cb1c2bb10573 WatchSource:0}: Error finding container 18ae576d98fdf0d4f113fb801cd36c378ce189d1f4c6793c3548cb1c2bb10573: Status 404 returned error can't find the container with id 18ae576d98fdf0d4f113fb801cd36c378ce189d1f4c6793c3548cb1c2bb10573 Nov 03 22:50:37.071207 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.071113 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr"] Nov 03 22:50:37.096311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.090573 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-588c64a2-wg7kr"] Nov 03 22:50:37.289980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.289935 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" event={"ID":"78d5786e-46b8-4ae6-957c-ccef18eb39b3","Type":"ContainerStarted","Data":"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9"} Nov 03 22:50:37.289980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.290026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" event={"ID":"78d5786e-46b8-4ae6-957c-ccef18eb39b3","Type":"ContainerStarted","Data":"41a8e04a5e2a53049a0d65b0d199821fcbd7f40ec042814e169ef8c157e8415d"} Nov 03 22:50:37.290260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.290065 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:37.292095 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.292062 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" event={"ID":"cd44189a-405c-4974-b08b-e71ebb0ceb20","Type":"ContainerStarted","Data":"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7"} Nov 03 22:50:37.292227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.292114 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" event={"ID":"cd44189a-405c-4974-b08b-e71ebb0ceb20","Type":"ContainerStarted","Data":"18ae576d98fdf0d4f113fb801cd36c378ce189d1f4c6793c3548cb1c2bb10573"} Nov 03 22:50:37.292227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.292135 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:37.446389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.446333 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" podStartSLOduration=2.446318071 podStartE2EDuration="2.446318071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:37.441818183 +0000 UTC m=+1711.179162410" watchObservedRunningTime="2026-11-03 22:50:37.446318071 +0000 UTC m=+1711.183662296" Nov 03 22:50:37.608727 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:37.608657 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" podStartSLOduration=2.608637952 podStartE2EDuration="2.608637952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:37.607339734 +0000 UTC m=+1711.344683959" watchObservedRunningTime="2026-11-03 22:50:37.608637952 +0000 UTC m=+1711.345982180" Nov 03 22:50:38.411289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:38.411254 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a877ed-883c-4054-a214-097afb14700d" path="/var/lib/kubelet/pods/83a877ed-883c-4054-a214-097afb14700d/volumes" Nov 03 22:50:43.217507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:43.217476 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:50:43.217960 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:43.217560 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:50:43.354287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:43.354223 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:50:43.354603 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:43.354559 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:50:46.401444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.401342 5051 generic.go:356] "Generic (PLEG): container finished" podID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerID="e6a1df71ba436dad6997906d4ff354cc714d77c8950f4fef071ab22f85336432" exitCode=0 Nov 03 22:50:46.410938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.410895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerDied","Data":"e6a1df71ba436dad6997906d4ff354cc714d77c8950f4fef071ab22f85336432"} Nov 03 22:50:46.411393 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.411372 5051 scope.go:122] "RemoveContainer" containerID="e6a1df71ba436dad6997906d4ff354cc714d77c8950f4fef071ab22f85336432" Nov 03 22:50:46.412966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.412933 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:46.413132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.413036 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:46.413132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:46.413090 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:47.419679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:47.419640 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerStarted","Data":"ffcccc4c75de48c81b7d33c5f4daedb12c89293b5b52f4247adefdf3707d6ec4"} Nov 03 22:50:47.943460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:47.943425 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:50:47.943663 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:47.943640 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:50:47.943711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:47.943698 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 22:50:48.050120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.050087 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b"] Nov 03 22:50:48.054294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.054266 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b8743b6-wdq5b"] Nov 03 22:50:48.308126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.308037 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:50:48.308285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.308154 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:50:48.421408 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.421367 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d5c789-7603-4e49-b0f4-1d0835b6dc3c" path="/var/lib/kubelet/pods/48d5c789-7603-4e49-b0f4-1d0835b6dc3c/volumes" Nov 03 22:50:48.944635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.944598 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:50:48.944953 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:48.944656 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 22:50:49.943896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:49.943854 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:50:49.944381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:49.943933 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 22:50:50.944622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:50.944582 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:50:50.945118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:50.944662 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 22:50:51.627914 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:50:51.627873 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:50:51.944196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:51.944107 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 22:50:51.944196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:51.944162 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 22:50:52.960196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:52.960163 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:50:52.960751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:52.960729 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:50:52.961595 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:50:52.961572 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 22:51:08.630886 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.630856 5051 scope.go:122] "RemoveContainer" containerID="8989e5ab3d8439a62fa61004225948d39b730f3163fed92b0977d0f3729c748b" Nov 03 22:51:08.644607 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.644583 5051 scope.go:122] "RemoveContainer" containerID="560ee820cc373b414107b8587f3c30685e2c689665db7e1c46733a3203fbbd71" Nov 03 22:51:08.655725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.655702 5051 scope.go:122] "RemoveContainer" containerID="6519e03646f38fb3fbeee9b810aedbe7e7b3b36335fe2f0d3e0effe25794fbab" Nov 03 22:51:08.681694 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.681660 5051 scope.go:122] "RemoveContainer" containerID="e153a47784d5e48b22672669d0e4868d4ee93fabb45c001a7ddb7c1ca5182a3a" Nov 03 22:51:08.693572 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.693539 5051 scope.go:122] "RemoveContainer" containerID="f45fcb00bb04e3397c813f88cde1fcff0a99bbe70101bf1322fe4167fdeb160f" Nov 03 22:51:08.708277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.708254 5051 scope.go:122] "RemoveContainer" containerID="6e4f531cd42ab83e66d8770fc832f6abd6c743bb506866f4dad8cca92478b6ce" Nov 03 22:51:08.719870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.719847 5051 scope.go:122] "RemoveContainer" containerID="06c32d764c98da70c14968ae3dad23184887b7f859496a184da3ae497cc09760" Nov 03 22:51:08.731264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.731240 5051 scope.go:122] "RemoveContainer" containerID="161c4f8856886f66ff3f4a0ff3677ed4ce6d2f89c25aabdade906211f126f543" Nov 03 22:51:08.745699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.745672 5051 scope.go:122] "RemoveContainer" containerID="c6aa075ab131853e61674c21e4303fa70789a57f88e6b4003f1b725fc60ca018" Nov 03 22:51:08.756890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.756867 5051 scope.go:122] "RemoveContainer" containerID="3cba564b67dc7bd7787e9b19169384a4803da9f2a66052c8eb3fff575a393f8b" Nov 03 22:51:08.768259 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.768235 5051 scope.go:122] "RemoveContainer" containerID="50ccee7571bdada1f0fa45503aae1d2bcc0ffd7dad67c2833b45f8659d42f2d8" Nov 03 22:51:08.782032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.782009 5051 scope.go:122] "RemoveContainer" containerID="a59dfb75d1584216cb5ad4c30ea366cf4403f00c8c2a8657c083b12a5d3efcd5" Nov 03 22:51:08.794720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.794690 5051 scope.go:122] "RemoveContainer" containerID="cff389e2e1e7d532ac9852ba464d5353566c85f5541994baae4388f4bb90c694" Nov 03 22:51:08.807895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:08.807875 5051 scope.go:122] "RemoveContainer" containerID="6b1d915d5753083642c3c182cd6d21bc139c8c3f9f1feab1e953f01bf8758eb9" Nov 03 22:51:13.703835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.703798 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 22:51:13.711067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.711043 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.775679 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.775640 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 22:51:13.810974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.810930 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a9ab23c4\"/\"openshift-service-ca.crt\"" Nov 03 22:51:13.811294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.811100 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a9ab23c4\"/\"kube-root-ca.crt\"" Nov 03 22:51:13.814973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.814947 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a9ab23c4\"/\"e2e-lifecycle-model-a9ab23c4-kserve-self-signed-certs\"" Nov 03 22:51:13.815120 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.814951 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a9ab23c4\"/\"default-dockercfg-sf9dq\"" Nov 03 22:51:13.833206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833165 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.833392 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833366 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.833460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833421 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fwc\" (UniqueName: \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.833645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833632 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.833682 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833671 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.833724 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.833701 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.935638 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935596 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.935809 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935656 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fwc\" (UniqueName: \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.935902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935825 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.935902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935866 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.935902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935890 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.936102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.935912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.936265 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.936238 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.936340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.936259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.936340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.936305 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.938252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.938230 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:13.938491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:13.938473 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:14.005623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.005508 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fwc\" (UniqueName: \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\") pod \"e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:14.027100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.026727 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:14.456445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.456409 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 22:51:14.713212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.713119 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" event={"ID":"94054aa8-acd2-4351-a7fb-bf663e5c2c2f","Type":"ContainerStarted","Data":"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f"} Nov 03 22:51:14.713655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.713636 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" event={"ID":"94054aa8-acd2-4351-a7fb-bf663e5c2c2f","Type":"ContainerStarted","Data":"599baa01b6cdd5c9576963801de853a6e5c3de891a6fe9a0009f1d88a135daf2"} Nov 03 22:51:14.832335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:14.832283 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" podStartSLOduration=1.832269409 podStartE2EDuration="1.832269409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:14.828164007 +0000 UTC m=+1748.565508232" watchObservedRunningTime="2026-11-03 22:51:14.832269409 +0000 UTC m=+1748.569613665" Nov 03 22:51:15.027576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:15.027494 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:21.626471 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:51:21.626426 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:51:24.027383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:24.027345 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:24.042515 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:24.042485 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:24.813332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:24.813305 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 22:51:31.805649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:31.805613 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 22:51:31.845831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:31.845803 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:31.932203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:31.932172 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 22:51:32.010587 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.010554 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-433db8-dockercfg-xr684\"" Nov 03 22:51:32.031131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.011235 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-433db8\"" Nov 03 22:51:32.031131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.013599 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v5r\" (UniqueName: \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.031382 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.031305 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.132885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.132853 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v5r\" (UniqueName: \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.133116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.132951 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.133690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.133668 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.205759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.205725 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v5r\" (UniqueName: \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\") pod \"payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.332574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.332539 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:32.723118 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:51:32.723088 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19988de7_6b9c_49d8_918d_c77cdf727daa.slice/crio-97cfa99a3337b85a0da0fb497d9e84f061e0a7ea7064fe606fea67179e2a81c9 WatchSource:0}: Error finding container 97cfa99a3337b85a0da0fb497d9e84f061e0a7ea7064fe606fea67179e2a81c9: Status 404 returned error can't find the container with id 97cfa99a3337b85a0da0fb497d9e84f061e0a7ea7064fe606fea67179e2a81c9 Nov 03 22:51:32.725711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.725689 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 22:51:32.964424 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.964375 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" event={"ID":"19988de7-6b9c-49d8-918d-c77cdf727daa","Type":"ContainerStarted","Data":"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123"} Nov 03 22:51:32.964906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.964439 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" event={"ID":"19988de7-6b9c-49d8-918d-c77cdf727daa","Type":"ContainerStarted","Data":"97cfa99a3337b85a0da0fb497d9e84f061e0a7ea7064fe606fea67179e2a81c9"} Nov 03 22:51:32.964906 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:32.964622 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:33.087955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:33.087848 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" podStartSLOduration=2.087835713 podStartE2EDuration="2.087835713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:33.086348002 +0000 UTC m=+1766.823692227" watchObservedRunningTime="2026-11-03 22:51:33.087835713 +0000 UTC m=+1766.825179948" Nov 03 22:51:35.056685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:35.056652 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf"] Nov 03 22:51:35.097751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:35.097722 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36264199-lpxqf"] Nov 03 22:51:36.410221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:36.410188 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1a3458-c9df-47ce-abf2-a2f8b0094092" path="/var/lib/kubelet/pods/8b1a3458-c9df-47ce-abf2-a2f8b0094092/volumes" Nov 03 22:51:39.206826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:39.206782 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:51:39.207402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:39.207141 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" containerID="cri-o://9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250" gracePeriod=30 Nov 03 22:51:43.519636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:43.519599 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:43.520081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:43.519674 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:43.995545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:43.995515 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 22:51:44.544358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.544334 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:51:44.680825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.680795 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.680910 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.680957 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.680982 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681029 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681075 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681131 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681244 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.681463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681426 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\") pod \"37830e06-896a-49da-88d9-a8eadc034556\" (UID: \"37830e06-896a-49da-88d9-a8eadc034556\") " Nov 03 22:51:44.682281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.681817 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:44.682281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682088 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:44.682281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682267 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:44.682586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682560 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.682667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682600 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.682667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682615 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.682667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682562 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:44.682831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.682812 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:44.706179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.706121 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:44.706179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.706134 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:44.706179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.706173 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:44.706416 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.706200 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58" pod "37830e06-896a-49da-88d9-a8eadc034556" (UID: "37830e06-896a-49da-88d9-a8eadc034556"). InnerVolumeSpecName "kube-api-access-ksf58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:44.784820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784790 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37830e06-896a-49da-88d9-a8eadc034556-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.784820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784819 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.785092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784832 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37830e06-896a-49da-88d9-a8eadc034556-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.785092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784841 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.785092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784849 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37830e06-896a-49da-88d9-a8eadc034556-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:44.785092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:44.784857 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksf58\" (UniqueName: \"kubernetes.io/projected/37830e06-896a-49da-88d9-a8eadc034556-kube-api-access-ksf58\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:45.131067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.131037 5051 generic.go:356] "Generic (PLEG): container finished" podID="37830e06-896a-49da-88d9-a8eadc034556" containerID="9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250" exitCode=0 Nov 03 22:51:45.131254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.131086 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" Nov 03 22:51:45.131254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.131130 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" event={"ID":"37830e06-896a-49da-88d9-a8eadc034556","Type":"ContainerDied","Data":"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250"} Nov 03 22:51:45.131254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.131188 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv" event={"ID":"37830e06-896a-49da-88d9-a8eadc034556","Type":"ContainerDied","Data":"a8b950b6cd19963ca2e1a16a273bd91acfcb431326a195158b0ba33c060ca827"} Nov 03 22:51:45.131254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.131211 5051 scope.go:122] "RemoveContainer" containerID="9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250" Nov 03 22:51:45.143019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.143003 5051 scope.go:122] "RemoveContainer" containerID="9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250" Nov 03 22:51:45.143318 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:51:45.143297 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250\": container with ID starting with 9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250 not found: ID does not exist" containerID="9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250" Nov 03 22:51:45.143374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.143326 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250"} err="failed to get container status \"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250\": rpc error: code = NotFound desc = could not find container \"9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250\": container with ID starting with 9fda7443c3709005f3fefa15cbc42c625bf8f7c341a899fb4dbc5f9505a77250 not found: ID does not exist" Nov 03 22:51:45.301302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.301262 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:51:45.348267 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:45.348236 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-nprdv"] Nov 03 22:51:46.410579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:46.410546 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37830e06-896a-49da-88d9-a8eadc034556" path="/var/lib/kubelet/pods/37830e06-896a-49da-88d9-a8eadc034556/volumes" Nov 03 22:51:51.635052 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:51:51.635005 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:51:54.049564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:54.049533 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6"] Nov 03 22:51:54.134648 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:54.134551 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2cfc60-8r5k6"] Nov 03 22:51:54.434646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:51:54.434606 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bee6c4-58c4-41d1-92c0-1215b5a1805e" path="/var/lib/kubelet/pods/36bee6c4-58c4-41d1-92c0-1215b5a1805e/volumes" Nov 03 22:52:00.206980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.206129 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:00.206980 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.206206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:00.310477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.310447 5051 generic.go:356] "Generic (PLEG): container finished" podID="83820620-a67c-4073-a7e7-3d05224575a8" containerID="b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557" exitCode=0 Nov 03 22:52:00.310683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.310532 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerDied","Data":"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557"} Nov 03 22:52:00.311084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.311063 5051 scope.go:122] "RemoveContainer" containerID="b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557" Nov 03 22:52:00.312419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312399 5051 generic.go:356] "Generic (PLEG): container finished" podID="c31cf560-8002-451f-bc14-e2f8792597d3" containerID="dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b" exitCode=0 Nov 03 22:52:00.312499 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312426 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerDied","Data":"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b"} Nov 03 22:52:00.312801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312788 5051 scope.go:122] "RemoveContainer" containerID="dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b" Nov 03 22:52:00.312848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312806 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:00.312885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312870 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:00.312919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.312904 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:00.314542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.314511 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:00.314626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.314610 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:00.314674 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:00.314660 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:01.328665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:01.328623 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerStarted","Data":"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15"} Nov 03 22:52:01.331294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:01.331272 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerStarted","Data":"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de"} Nov 03 22:52:01.754575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:01.754544 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:52:01.754855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:01.754832 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 22:52:01.754928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:01.754877 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 22:52:02.121079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:02.101204 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:52:02.121079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:02.101331 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:02.121079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:02.101374 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:02.754242 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:02.754197 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 22:52:02.754635 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:02.754249 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 22:52:03.102503 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:03.102469 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:03.102790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:03.102763 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:03.754140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:03.754106 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 22:52:03.754318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:03.754159 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 22:52:04.112072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:04.109167 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:04.112072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:04.109241 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:04.754413 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:04.754378 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 22:52:04.754805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:04.754425 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 22:52:05.106012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:05.103467 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:05.106012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:05.103531 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:05.754949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:05.754915 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 22:52:05.755543 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:05.755515 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 22:52:06.102070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.098186 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 22:52:06.102070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.098247 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 22:52:06.782576 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.782548 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:52:06.783565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.783539 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:52:06.784913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.784892 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:52:06.831353 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.831315 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:52:06.831868 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:06.831848 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:52:07.103485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:07.103447 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:52:07.104767 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:07.104747 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:52:07.105545 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:07.105518 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 22:52:09.052975 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:09.052939 5051 scope.go:122] "RemoveContainer" containerID="219035f316d0a413d40d7cd8f37fa448ad0f56091160743f079e4942b2d3c0bf" Nov 03 22:52:09.065435 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:09.065238 5051 scope.go:122] "RemoveContainer" containerID="3ac816f04c6d7afd03fe9d00c4e48d941422aa0ad449c1004536ff5e6064d5d6" Nov 03 22:52:09.079582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:09.079561 5051 scope.go:122] "RemoveContainer" containerID="3ea2b96071a4a8ac8cabd8667764826f7d000c09e51b562474a42f220f102f61" Nov 03 22:52:15.664982 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.664946 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6"] Nov 03 22:52:15.665770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.665753 5051 cpu_manager.go:414] "containerMap: removing container" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" Nov 03 22:52:15.665828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.665772 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" Nov 03 22:52:15.665892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.665883 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="37830e06-896a-49da-88d9-a8eadc034556" containerName="istio-proxy" Nov 03 22:52:15.669615 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.669599 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.727165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.727130 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6"] Nov 03 22:52:15.771322 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.771294 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:15.771495 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.771430 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:52:15.785996 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.785965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.786156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.786026 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrkx\" (UniqueName: \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.786235 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.786180 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.887334 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.887300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.887555 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.887538 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.887646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.887590 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrkx\" (UniqueName: \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.888370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.888337 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.889946 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.889923 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.948511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.948452 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrkx\" (UniqueName: \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\") pod \"maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:15.987115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:15.987093 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:16.384858 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:16.384813 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6"] Nov 03 22:52:16.385324 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:52:16.385299 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45635b4_635e_4515_846a_4b417bfa89ce.slice/crio-4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907 WatchSource:0}: Error finding container 4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907: Status 404 returned error can't find the container with id 4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907 Nov 03 22:52:16.556017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:16.553522 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" event={"ID":"e45635b4-635e-4515-846a-4b417bfa89ce","Type":"ContainerStarted","Data":"4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907"} Nov 03 22:52:17.562972 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:17.562944 5051 generic.go:356] "Generic (PLEG): container finished" podID="e45635b4-635e-4515-846a-4b417bfa89ce" containerID="88a57f06b4851aa05950aec906846042df65d2641a714b42074cbda07f269708" exitCode=0 Nov 03 22:52:17.563366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:17.563050 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" event={"ID":"e45635b4-635e-4515-846a-4b417bfa89ce","Type":"ContainerDied","Data":"88a57f06b4851aa05950aec906846042df65d2641a714b42074cbda07f269708"} Nov 03 22:52:18.740323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.740297 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:18.872352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.872326 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\") pod \"e45635b4-635e-4515-846a-4b417bfa89ce\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " Nov 03 22:52:18.872540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.872524 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\" (UniqueName: \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\") pod \"e45635b4-635e-4515-846a-4b417bfa89ce\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " Nov 03 22:52:18.872685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.872666 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\") pod \"e45635b4-635e-4515-846a-4b417bfa89ce\" (UID: \"e45635b4-635e-4515-846a-4b417bfa89ce\") " Nov 03 22:52:18.872765 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.872705 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca" pod "e45635b4-635e-4515-846a-4b417bfa89ce" (UID: "e45635b4-635e-4515-846a-4b417bfa89ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:18.874041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.873960 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:18.875072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.875042 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx" pod "e45635b4-635e-4515-846a-4b417bfa89ce" (UID: "e45635b4-635e-4515-846a-4b417bfa89ce"). InnerVolumeSpecName "kube-api-access-dnrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:18.887466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.887433 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca" pod "e45635b4-635e-4515-846a-4b417bfa89ce" (UID: "e45635b4-635e-4515-846a-4b417bfa89ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:18.975122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.975079 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrkx\" (UniqueName: \"kubernetes.io/projected/e45635b4-635e-4515-846a-4b417bfa89ce-kube-api-access-dnrkx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:18.975122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:18.975110 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45635b4-635e-4515-846a-4b417bfa89ce-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:19.581973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:19.581942 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" Nov 03 22:52:19.581973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:19.581953 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6" event={"ID":"e45635b4-635e-4515-846a-4b417bfa89ce","Type":"ContainerDied","Data":"4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907"} Nov 03 22:52:19.582218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:19.582027 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4daed6e0b5db2f65dc595069e1fb6a090bbd393d6ca9f9ea9db146ffadf20907" Nov 03 22:52:21.626630 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:52:21.626597 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:52:25.212315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.212276 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:52:25.213528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.213505 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e45635b4-635e-4515-846a-4b417bfa89ce" containerName="revoke-keys" Nov 03 22:52:25.213528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.213530 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45635b4-635e-4515-846a-4b417bfa89ce" containerName="revoke-keys" Nov 03 22:52:25.213729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.213716 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45635b4-635e-4515-846a-4b417bfa89ce" containerName="revoke-keys" Nov 03 22:52:25.217256 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.217233 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.281914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.281877 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-efce83\"" Nov 03 22:52:25.332189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.332158 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-efce83-dockercfg-2vsd9\"" Nov 03 22:52:25.338854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.338810 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:52:25.342023 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.340663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.342281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.342250 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzqg\" (UniqueName: \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.433016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.423782 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:52:25.471073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.467343 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.473319 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.473287 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzqg\" (UniqueName: \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.476721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.476696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.477629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.477611 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:52:25.477865 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.477819 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.543398 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.543362 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzqg\" (UniqueName: \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\") pod \"payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.575216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.575191 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9mq\" (UniqueName: \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.575472 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.575457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.677759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.677723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9mq\" (UniqueName: \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.677933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.677856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.678775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.678749 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.738323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.738260 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9mq\" (UniqueName: \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\") pod \"payload-processing-e2e-api-a-efce83-84867f4d46-24l2f\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:25.834854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.834826 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:25.851696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:25.851672 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:26.248020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.247971 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:52:26.267587 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:52:26.267555 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6c9654_fe35_41d7_bffc_cfd9f4f3d3fd.slice/crio-0f272ae2fa1f33a56ef46301179ec987651c87f34f24a7d4889ec588dd67847d WatchSource:0}: Error finding container 0f272ae2fa1f33a56ef46301179ec987651c87f34f24a7d4889ec588dd67847d: Status 404 returned error can't find the container with id 0f272ae2fa1f33a56ef46301179ec987651c87f34f24a7d4889ec588dd67847d Nov 03 22:52:26.299087 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.299062 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:52:26.305635 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:52:26.305608 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb94bb4_0312_4293_ae92_b433eb28f28d.slice/crio-ddf071dc6520690e34e1eec76e5393d98d0dc27486953e6a832508dfd97b482c WatchSource:0}: Error finding container ddf071dc6520690e34e1eec76e5393d98d0dc27486953e6a832508dfd97b482c: Status 404 returned error can't find the container with id ddf071dc6520690e34e1eec76e5393d98d0dc27486953e6a832508dfd97b482c Nov 03 22:52:26.643606 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.643572 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" event={"ID":"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd","Type":"ContainerStarted","Data":"9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e"} Nov 03 22:52:26.643771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.643628 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" event={"ID":"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd","Type":"ContainerStarted","Data":"0f272ae2fa1f33a56ef46301179ec987651c87f34f24a7d4889ec588dd67847d"} Nov 03 22:52:26.643771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.643652 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:26.645191 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.645168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" event={"ID":"6cb94bb4-0312-4293-ae92-b433eb28f28d","Type":"ContainerStarted","Data":"dd7f36507a5c2820bf96a9387e34d00dd641d245549a9baa8fe9c08a66cb6066"} Nov 03 22:52:26.645312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.645211 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" event={"ID":"6cb94bb4-0312-4293-ae92-b433eb28f28d","Type":"ContainerStarted","Data":"ddf071dc6520690e34e1eec76e5393d98d0dc27486953e6a832508dfd97b482c"} Nov 03 22:52:26.645312 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.645233 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:26.769657 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.769601 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" podStartSLOduration=1.769551713 podStartE2EDuration="1.769551713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:26.766170622 +0000 UTC m=+1820.503514848" watchObservedRunningTime="2026-11-03 22:52:26.769551713 +0000 UTC m=+1820.506895937" Nov 03 22:52:26.899151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:26.891897 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" podStartSLOduration=1.8918783289999999 podStartE2EDuration="1.891878329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:26.884261098 +0000 UTC m=+1820.621605324" watchObservedRunningTime="2026-11-03 22:52:26.891878329 +0000 UTC m=+1820.629222554" Nov 03 22:52:28.419298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:28.416352 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:52:33.416870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.416800 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" podUID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" containerName="payload-pre-processing" containerID="cri-o://69d9506e035f6eee2a658fdcab60bc86f2828debb19f7c2f5b301e82954262d5" gracePeriod=25 Nov 03 22:52:33.734068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.733805 5051 generic.go:356] "Generic (PLEG): container finished" podID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" containerID="69d9506e035f6eee2a658fdcab60bc86f2828debb19f7c2f5b301e82954262d5" exitCode=0 Nov 03 22:52:33.734068 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.734045 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" event={"ID":"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12","Type":"ContainerDied","Data":"69d9506e035f6eee2a658fdcab60bc86f2828debb19f7c2f5b301e82954262d5"} Nov 03 22:52:33.837158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.837128 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:52:33.855302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.854721 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\" (UniqueName: \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\") pod \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " Nov 03 22:52:33.855566 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.855547 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\") pod \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\" (UID: \"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12\") " Nov 03 22:52:33.864177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.856847 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume" pod "1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" (UID: "1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:33.864177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.857561 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:33.864177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.858046 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b" pod "1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" (UID: "1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12"). InnerVolumeSpecName "kube-api-access-d459b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:33.959391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:33.959355 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d459b\" (UniqueName: \"kubernetes.io/projected/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12-kube-api-access-d459b\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:34.798093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:34.789408 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" event={"ID":"1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12","Type":"ContainerDied","Data":"b2d773ec8425edace6231c7ebb76cd138d27ce5189b78e02b877dd8031810206"} Nov 03 22:52:34.798093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:34.789494 5051 scope.go:122] "RemoveContainer" containerID="69d9506e035f6eee2a658fdcab60bc86f2828debb19f7c2f5b301e82954262d5" Nov 03 22:52:34.798093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:34.789691 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4" Nov 03 22:52:34.958597 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:34.958565 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:52:35.008232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:35.008205 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ebffae22-76f58b75cf-kmqd4"] Nov 03 22:52:36.409934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:36.409906 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" path="/var/lib/kubelet/pods/1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12/volumes" Nov 03 22:52:37.655460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:37.655430 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:52:37.655862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:37.655507 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:52:41.618085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.618051 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:52:41.619795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.619768 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" containerName="payload-pre-processing" Nov 03 22:52:41.620035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.620018 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" containerName="payload-pre-processing" Nov 03 22:52:41.620410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.620395 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba3a7fc-dea8-4d2e-ad36-4dff4db7af12" containerName="payload-pre-processing" Nov 03 22:52:41.635101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.635076 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.667410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.667373 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:52:41.735090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.735058 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.735388 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.735367 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glltk\" (UniqueName: \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.743361 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.743331 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-26444f\"" Nov 03 22:52:41.744363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.744297 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-26444f-dockercfg-59m5j\"" Nov 03 22:52:41.837673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.836520 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glltk\" (UniqueName: \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.837673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.836683 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.837673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.837606 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.906059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.905932 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glltk\" (UniqueName: \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\") pod \"payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:41.965048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:41.964758 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:42.390164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:42.390122 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:52:42.912911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:42.912880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" event={"ID":"17fdc5a3-17f3-425f-bc92-8aa731556312","Type":"ContainerStarted","Data":"971e960a4a897d7033328c03664a4df8366638549d885f1144c812f90e7f9fc7"} Nov 03 22:52:42.913389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:42.912934 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" event={"ID":"17fdc5a3-17f3-425f-bc92-8aa731556312","Type":"ContainerStarted","Data":"b930da756d221aac8e7884d551fd9d66d342cf0172426ae41950783b435655e7"} Nov 03 22:52:42.913389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:42.912970 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:52:43.027523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:43.027465 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" podStartSLOduration=2.027448148 podStartE2EDuration="2.027448148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:43.026959449 +0000 UTC m=+1836.764303675" watchObservedRunningTime="2026-11-03 22:52:43.027448148 +0000 UTC m=+1836.764792371" Nov 03 22:52:51.627664 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:52:51.627582 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:52:53.924777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:52:53.924750 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:53:07.612186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.611700 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:53:07.621386 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.621357 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.726950 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.726910 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:53:07.727300 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.727241 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0406a8\"" Nov 03 22:53:07.727536 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.727518 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-0406a8-dockercfg-cf699\"" Nov 03 22:53:07.767206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.767098 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.768784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.768758 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd88\" (UniqueName: \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.892730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.892642 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.893086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.893063 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd88\" (UniqueName: \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.902629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.893967 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:07.954897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:07.954870 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd88\" (UniqueName: \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\") pod \"payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:08.239391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:08.239308 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:08.625410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:08.625377 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:53:08.626385 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:53:08.626360 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02627d5e_2e7b_4f2e_b2bf_c49b18248a8f.slice/crio-bd9aa2ce1b6933656887ae53cddd526cd1079f88c4820df63216a89e03af33fd WatchSource:0}: Error finding container bd9aa2ce1b6933656887ae53cddd526cd1079f88c4820df63216a89e03af33fd: Status 404 returned error can't find the container with id bd9aa2ce1b6933656887ae53cddd526cd1079f88c4820df63216a89e03af33fd Nov 03 22:53:09.124451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.124418 5051 scope.go:122] "RemoveContainer" containerID="5d7bd00f634713909e658e861b230485c3e208a3cf95b8847e7c766fabd53b19" Nov 03 22:53:09.135362 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.135337 5051 scope.go:122] "RemoveContainer" containerID="642795a4e574d8396b03ad7cd41d64280c415896a0f8fca6429dccf52fee8d01" Nov 03 22:53:09.149289 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.149254 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" event={"ID":"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f","Type":"ContainerStarted","Data":"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285"} Nov 03 22:53:09.149421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.149322 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" event={"ID":"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f","Type":"ContainerStarted","Data":"bd9aa2ce1b6933656887ae53cddd526cd1079f88c4820df63216a89e03af33fd"} Nov 03 22:53:09.149421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.149356 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:09.152400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.152378 5051 scope.go:122] "RemoveContainer" containerID="2fb2c311643ff1ffb6580ef1f89fe73f5bc6b85f2189c344662dc8b62c4a9386" Nov 03 22:53:09.163612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.163590 5051 scope.go:122] "RemoveContainer" containerID="524a35f6a9eaf13fddc2cda89de0f139badadc65352edad7570e68c641dce41a" Nov 03 22:53:09.267452 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:09.267407 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" podStartSLOduration=2.267390464 podStartE2EDuration="2.267390464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:09.264095178 +0000 UTC m=+1863.001439402" watchObservedRunningTime="2026-11-03 22:53:09.267390464 +0000 UTC m=+1863.004734688" Nov 03 22:53:13.642074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:13.642039 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6"] Nov 03 22:53:13.690528 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:13.690499 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebffae22-xlqz6"] Nov 03 22:53:14.411247 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:14.411215 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45635b4-635e-4515-846a-4b417bfa89ce" path="/var/lib/kubelet/pods/e45635b4-635e-4515-846a-4b417bfa89ce/volumes" Nov 03 22:53:20.162325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:20.162296 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:53:21.625037 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:53:21.624970 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:53:21.754553 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:21.754525 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:53:21.754751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:21.754730 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" containerID="cri-o://62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d" gracePeriod=30 Nov 03 22:53:25.707901 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:25.707867 5051 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:25.708492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:25.707939 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:27.136727 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.136682 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:53:27.227372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227326 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227427 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227469 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227534 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227589 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227605 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.227759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227667 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227828 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.227921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227875 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.227921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227907 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.228163 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227959 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\") pod \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\" (UID: \"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1\") " Nov 03 22:53:27.228163 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.227967 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.228163 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.228061 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.228340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.228317 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:27.229043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.228970 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.229158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.229103 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.229158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.229126 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.229158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.229140 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.229158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.229153 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.231364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.231341 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:27.231502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.231354 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.231589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.231552 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:27.232665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.232647 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch" pod "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" (UID: "b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1"). InnerVolumeSpecName "kube-api-access-pcnch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:27.331081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.331005 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.331081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.331035 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcnch\" (UniqueName: \"kubernetes.io/projected/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-kube-api-access-pcnch\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.331081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.331045 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.331081 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.331055 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.530166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.530133 5051 generic.go:356] "Generic (PLEG): container finished" podID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerID="62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d" exitCode=0 Nov 03 22:53:27.530339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.530181 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" Nov 03 22:53:27.530339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.530211 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerDied","Data":"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d"} Nov 03 22:53:27.530339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.530269 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7" event={"ID":"b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1","Type":"ContainerDied","Data":"fa1fbcd0b2fd76f65ac6edc821581fb99438b7e4daa07096cc99d3e607e6e059"} Nov 03 22:53:27.530339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.530300 5051 scope.go:122] "RemoveContainer" containerID="62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d" Nov 03 22:53:27.542350 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.542331 5051 scope.go:122] "RemoveContainer" containerID="bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711" Nov 03 22:53:27.553098 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.553082 5051 scope.go:122] "RemoveContainer" containerID="62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d" Nov 03 22:53:27.553360 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:53:27.553344 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d\": container with ID starting with 62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d not found: ID does not exist" containerID="62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d" Nov 03 22:53:27.553410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.553368 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d"} err="failed to get container status \"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d\": rpc error: code = NotFound desc = could not find container \"62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d\": container with ID starting with 62eefed608876dfd0eddda9f0bd220b483884f68d2e6d527d895d28e9bde204d not found: ID does not exist" Nov 03 22:53:27.553410 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.553386 5051 scope.go:122] "RemoveContainer" containerID="bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711" Nov 03 22:53:27.553614 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:53:27.553596 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711\": container with ID starting with bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711 not found: ID does not exist" containerID="bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711" Nov 03 22:53:27.553655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.553618 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711"} err="failed to get container status \"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711\": rpc error: code = NotFound desc = could not find container \"bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711\": container with ID starting with bcff0a301daad0ad4007f588e1c3d2d690b40b59cfe74927028d716d846b5711 not found: ID does not exist" Nov 03 22:53:27.699961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.699931 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:53:27.746732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:27.746700 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-tg5g7"] Nov 03 22:53:28.410916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:28.410887 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" path="/var/lib/kubelet/pods/b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1/volumes" Nov 03 22:53:32.694795 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.694761 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:53:32.695929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.695900 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.695929 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.695925 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.696182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.695960 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.696182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.695969 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.696182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.696167 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.696182 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.696181 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a7cc4e-ed8b-4a83-b7e1-a86f794e5be1" containerName="istio-proxy" Nov 03 22:53:32.701777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.701756 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.756427 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.756397 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:53:32.802285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.802252 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a91d2d\"" Nov 03 22:53:32.804041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.803974 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a91d2d-dockercfg-rw457\"" Nov 03 22:53:32.828041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.824751 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q2p\" (UniqueName: \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.828041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.824930 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.927855 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.927797 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.931008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.930954 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q2p\" (UniqueName: \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.932221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.932186 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:32.941490 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.941452 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:53:32.948257 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.947678 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:32.953782 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.953739 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:53:32.999074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:32.997219 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q2p\" (UniqueName: \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\") pod \"payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:33.019884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.019858 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:33.032968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.032926 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.033269 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.033244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nfm\" (UniqueName: \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.135194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.135150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nfm\" (UniqueName: \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.135370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.135280 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.136465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.136440 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.197094 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.197068 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nfm\" (UniqueName: \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\") pod \"payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.272973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.272906 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:33.412039 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:53:33.411960 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99ec5e_7270_4561_9e62_15b3950eb8c9.slice/crio-4c24d0939df72e217ca330323d65806998cc7453df4bcd5494fc66aa44feefee WatchSource:0}: Error finding container 4c24d0939df72e217ca330323d65806998cc7453df4bcd5494fc66aa44feefee: Status 404 returned error can't find the container with id 4c24d0939df72e217ca330323d65806998cc7453df4bcd5494fc66aa44feefee Nov 03 22:53:33.412477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.412449 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:53:33.585119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.585036 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" event={"ID":"ca99ec5e-7270-4561-9e62-15b3950eb8c9","Type":"ContainerStarted","Data":"fc6ff0a921593e13ddeb5d83e1239fdae4b7bee7a7ea86e3f4b17797a90ebb51"} Nov 03 22:53:33.585119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.585099 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" event={"ID":"ca99ec5e-7270-4561-9e62-15b3950eb8c9","Type":"ContainerStarted","Data":"4c24d0939df72e217ca330323d65806998cc7453df4bcd5494fc66aa44feefee"} Nov 03 22:53:33.585296 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.585125 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:33.687802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.686925 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:53:33.710034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:33.709946 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" podStartSLOduration=1.709931062 podStartE2EDuration="1.709931062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:33.700831222 +0000 UTC m=+1887.438175448" watchObservedRunningTime="2026-11-03 22:53:33.709931062 +0000 UTC m=+1887.447275285" Nov 03 22:53:34.595088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:34.595057 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" event={"ID":"081e4e07-8050-499a-8f1f-0771a825bf4d","Type":"ContainerStarted","Data":"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f"} Nov 03 22:53:34.595268 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:34.595111 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:34.595268 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:34.595130 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" event={"ID":"081e4e07-8050-499a-8f1f-0771a825bf4d","Type":"ContainerStarted","Data":"4ebb92c476c45b33ceaf3cd26db7b9c11ad33af701a0b01883e6b2ec586208c8"} Nov 03 22:53:34.715562 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:34.715517 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" podStartSLOduration=2.7155036089999998 podStartE2EDuration="2.715503609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:34.711597261 +0000 UTC m=+1888.448941485" watchObservedRunningTime="2026-11-03 22:53:34.715503609 +0000 UTC m=+1888.452847832" Nov 03 22:53:44.598636 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:44.598607 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:53:45.612160 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:53:45.612131 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:53:51.625878 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:53:51.625843 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:54:09.216717 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:09.216687 5051 scope.go:122] "RemoveContainer" containerID="815d2a4bea7a939f2765f5c220d4916dd00c174a0a7f6514f8bae63eac100cca" Nov 03 22:54:20.182885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.182810 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5"] Nov 03 22:54:20.188818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.188801 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.244496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.244470 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5"] Nov 03 22:54:20.288522 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.288485 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:20.288820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.288801 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:54:20.295509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.295488 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j99\" (UniqueName: \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.295602 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.295556 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.295656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.295633 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.396575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.396546 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j99\" (UniqueName: \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.396712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.396652 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.396712 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.396687 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.397320 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.397301 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.399403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.399384 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.457230 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.457169 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j99\" (UniqueName: \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.503792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.503770 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:20.892433 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:20.892402 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5"] Nov 03 22:54:20.892926 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:54:20.892903 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e3f2e5_abc3_41ec_a8bf_2abaa7d7aeee.slice/crio-8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08 WatchSource:0}: Error finding container 8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08: Status 404 returned error can't find the container with id 8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08 Nov 03 22:54:21.033448 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:21.033417 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" event={"ID":"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee","Type":"ContainerStarted","Data":"8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08"} Nov 03 22:54:21.623296 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:54:21.623264 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:54:22.043370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:22.043288 5051 generic.go:356] "Generic (PLEG): container finished" podID="d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" containerID="594c9caccd756562e7dd37aae64fd69f6cf927ff1a4c322b2eea38fdfc5afff6" exitCode=0 Nov 03 22:54:22.043532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:22.043362 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" event={"ID":"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee","Type":"ContainerDied","Data":"594c9caccd756562e7dd37aae64fd69f6cf927ff1a4c322b2eea38fdfc5afff6"} Nov 03 22:54:23.231336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.231311 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:23.344147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.344063 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\") pod \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " Nov 03 22:54:23.344299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.344268 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\" (UniqueName: \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\") pod \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " Nov 03 22:54:23.344357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.344309 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\") pod \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\" (UID: \"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee\") " Nov 03 22:54:23.344421 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.344394 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca" pod "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" (UID: "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:23.344949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.344879 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:23.346840 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.346809 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99" pod "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" (UID: "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee"). InnerVolumeSpecName "kube-api-access-r6j99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:23.346955 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.346843 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca" pod "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" (UID: "d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:23.446869 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.446836 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6j99\" (UniqueName: \"kubernetes.io/projected/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-kube-api-access-r6j99\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:23.446869 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:23.446870 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:24.061624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:24.061587 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" event={"ID":"d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee","Type":"ContainerDied","Data":"8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08"} Nov 03 22:54:24.061624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:24.061628 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5" Nov 03 22:54:24.061836 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:24.061644 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8528da4af8e7efa1b5deaebcc04b13f66e5583aab7c51a7d9b4f883f2859eb08" Nov 03 22:54:44.804188 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.804147 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s"] Nov 03 22:54:44.805069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.805052 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" containerName="revoke-keys" Nov 03 22:54:44.805069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.805070 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" containerName="revoke-keys" Nov 03 22:54:44.805202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.805193 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" containerName="revoke-keys" Nov 03 22:54:44.809202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.809184 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:44.864872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.864844 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s"] Nov 03 22:54:44.905781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.905744 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpz64\" (UniqueName: \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:44.905969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.905808 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:44.906156 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.906131 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:44.908656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.908631 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:44.908656 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:44.908647 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:54:45.007264 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.007230 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.007439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.007333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpz64\" (UniqueName: \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.007439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.007364 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.007867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.007844 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.009944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.009922 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.068527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.068463 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpz64\" (UniqueName: \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\") pod \"maas-api-revoke-sub-f2406dd7-cs96s\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.124564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.124534 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:45.510829 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:45.510799 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s"] Nov 03 22:54:45.511500 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:54:45.511469 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ff531e_f3c0_4004_a576_379b696e28fa.slice/crio-efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0 WatchSource:0}: Error finding container efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0: Status 404 returned error can't find the container with id efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0 Nov 03 22:54:46.376194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:46.376154 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" event={"ID":"97ff531e-f3c0-4004-a576-379b696e28fa","Type":"ContainerStarted","Data":"6ca7f6231e668a69da97697af81132e6954ddaad478102b25ed9465096ab20d5"} Nov 03 22:54:46.376621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:46.376251 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" event={"ID":"97ff531e-f3c0-4004-a576-379b696e28fa","Type":"ContainerStarted","Data":"efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0"} Nov 03 22:54:46.491570 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:46.491511 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" podStartSLOduration=2.4914923030000002 podStartE2EDuration="2.491492303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:46.489605771 +0000 UTC m=+1960.226949997" watchObservedRunningTime="2026-11-03 22:54:46.491492303 +0000 UTC m=+1960.228836523" Nov 03 22:54:47.385819 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:47.385784 5051 generic.go:356] "Generic (PLEG): container finished" podID="97ff531e-f3c0-4004-a576-379b696e28fa" containerID="6ca7f6231e668a69da97697af81132e6954ddaad478102b25ed9465096ab20d5" exitCode=0 Nov 03 22:54:47.386273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:47.385867 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" event={"ID":"97ff531e-f3c0-4004-a576-379b696e28fa","Type":"ContainerDied","Data":"6ca7f6231e668a69da97697af81132e6954ddaad478102b25ed9465096ab20d5"} Nov 03 22:54:48.577627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.577602 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:48.671928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.671842 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\") pod \"97ff531e-f3c0-4004-a576-379b696e28fa\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " Nov 03 22:54:48.672117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.672055 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\") pod \"97ff531e-f3c0-4004-a576-379b696e28fa\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " Nov 03 22:54:48.672192 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.672121 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\" (UniqueName: \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\") pod \"97ff531e-f3c0-4004-a576-379b696e28fa\" (UID: \"97ff531e-f3c0-4004-a576-379b696e28fa\") " Nov 03 22:54:48.672253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.672202 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca" pod "97ff531e-f3c0-4004-a576-379b696e28fa" (UID: "97ff531e-f3c0-4004-a576-379b696e28fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:48.672842 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.672821 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:48.674304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.674283 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca" pod "97ff531e-f3c0-4004-a576-379b696e28fa" (UID: "97ff531e-f3c0-4004-a576-379b696e28fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:48.674394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.674381 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64" pod "97ff531e-f3c0-4004-a576-379b696e28fa" (UID: "97ff531e-f3c0-4004-a576-379b696e28fa"). InnerVolumeSpecName "kube-api-access-xpz64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:48.774881 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.774851 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ff531e-f3c0-4004-a576-379b696e28fa-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:48.775064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:48.774880 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpz64\" (UniqueName: \"kubernetes.io/projected/97ff531e-f3c0-4004-a576-379b696e28fa-kube-api-access-xpz64\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:49.402864 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:49.402828 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" Nov 03 22:54:49.403108 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:49.402865 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s" event={"ID":"97ff531e-f3c0-4004-a576-379b696e28fa","Type":"ContainerDied","Data":"efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0"} Nov 03 22:54:49.403108 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:54:49.402920 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd824689cf507a75642cf7b1316d4891c586821bedd59d01ae341f06fd431f0" Nov 03 22:54:51.625938 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:54:51.625900 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:55:11.551890 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.551857 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4"] Nov 03 22:55:11.552718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.552697 5051 cpu_manager.go:414] "containerMap: removing container" podUID="97ff531e-f3c0-4004-a576-379b696e28fa" containerName="revoke-keys" Nov 03 22:55:11.552718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.552717 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ff531e-f3c0-4004-a576-379b696e28fa" containerName="revoke-keys" Nov 03 22:55:11.552880 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.552871 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ff531e-f3c0-4004-a576-379b696e28fa" containerName="revoke-keys" Nov 03 22:55:11.556672 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.556656 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.617033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.616959 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4"] Nov 03 22:55:11.654699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.654663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.654699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.654709 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4nc\" (UniqueName: \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.654962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.654837 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.659363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.659340 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:55:11.659540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.659527 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:55:11.756278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.756245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.756278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.756279 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4nc\" (UniqueName: \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.756650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.756334 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.757073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.757037 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.758808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.758787 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.817798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.817727 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4nc\" (UniqueName: \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\") pod \"maas-api-revoke-sub-96a62b53-jxbv4\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:11.870351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:11.870316 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:12.255784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:12.255738 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4"] Nov 03 22:55:12.256830 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:55:12.256803 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3e1282_5769_412e_9055_eeec76d312e9.slice/crio-27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3 WatchSource:0}: Error finding container 27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3: Status 404 returned error can't find the container with id 27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3 Nov 03 22:55:12.674834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:12.674800 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" event={"ID":"8d3e1282-5769-412e-9055-eeec76d312e9","Type":"ContainerStarted","Data":"27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3"} Nov 03 22:55:13.696826 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:13.696743 5051 generic.go:356] "Generic (PLEG): container finished" podID="8d3e1282-5769-412e-9055-eeec76d312e9" containerID="b8f39b1bfc1f369ba9aa83cec1607ff394b196facd0c0859f335014936dcf4e0" exitCode=0 Nov 03 22:55:13.697286 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:13.696836 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" event={"ID":"8d3e1282-5769-412e-9055-eeec76d312e9","Type":"ContainerDied","Data":"b8f39b1bfc1f369ba9aa83cec1607ff394b196facd0c0859f335014936dcf4e0"} Nov 03 22:55:14.889239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:14.889209 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:15.010687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.010603 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\") pod \"8d3e1282-5769-412e-9055-eeec76d312e9\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " Nov 03 22:55:15.010872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.010715 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\" (UniqueName: \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\") pod \"8d3e1282-5769-412e-9055-eeec76d312e9\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " Nov 03 22:55:15.010872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.010776 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\") pod \"8d3e1282-5769-412e-9055-eeec76d312e9\" (UID: \"8d3e1282-5769-412e-9055-eeec76d312e9\") " Nov 03 22:55:15.011016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.010939 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca" pod "8d3e1282-5769-412e-9055-eeec76d312e9" (UID: "8d3e1282-5769-412e-9055-eeec76d312e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:15.011288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.011263 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:15.013205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.013176 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca" pod "8d3e1282-5769-412e-9055-eeec76d312e9" (UID: "8d3e1282-5769-412e-9055-eeec76d312e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:15.013205 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.013187 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc" pod "8d3e1282-5769-412e-9055-eeec76d312e9" (UID: "8d3e1282-5769-412e-9055-eeec76d312e9"). InnerVolumeSpecName "kube-api-access-7x4nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:15.112945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.112911 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4nc\" (UniqueName: \"kubernetes.io/projected/8d3e1282-5769-412e-9055-eeec76d312e9-kube-api-access-7x4nc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:15.112945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.112943 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e1282-5769-412e-9055-eeec76d312e9-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:15.714967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.714935 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" Nov 03 22:55:15.715196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.715066 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4" event={"ID":"8d3e1282-5769-412e-9055-eeec76d312e9","Type":"ContainerDied","Data":"27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3"} Nov 03 22:55:15.715196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:15.715120 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27dc986c2435bc784c6308aaa13a6eecc1185313a670adf9ce55e70c60651bc3" Nov 03 22:55:19.962119 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:19.962081 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn"] Nov 03 22:55:19.962949 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:19.962930 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8d3e1282-5769-412e-9055-eeec76d312e9" containerName="revoke-keys" Nov 03 22:55:19.963012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:19.962951 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3e1282-5769-412e-9055-eeec76d312e9" containerName="revoke-keys" Nov 03 22:55:19.963112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:19.963102 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3e1282-5769-412e-9055-eeec76d312e9" containerName="revoke-keys" Nov 03 22:55:19.967278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:19.967258 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.019301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.019276 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:55:20.024623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.024600 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn"] Nov 03 22:55:20.068825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.068795 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:55:20.100411 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.100375 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk569\" (UniqueName: \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.100612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.100592 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.100710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.100650 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.202171 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.202129 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.202379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.202233 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.202451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.202434 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk569\" (UniqueName: \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.202951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.202922 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.204895 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.204872 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.264088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.264020 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk569\" (UniqueName: \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\") pod \"maas-api-revoke-keys-e2e-api-a-efce83-zfrcn\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.281866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.281837 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:20.667470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.667429 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn"] Nov 03 22:55:20.669261 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:55:20.669222 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66451bf_8657_4eb0_b592_b554812ae308.slice/crio-5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc WatchSource:0}: Error finding container 5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc: Status 404 returned error can't find the container with id 5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc Nov 03 22:55:20.771299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:20.771264 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" event={"ID":"f66451bf-8657-4eb0-b592-b554812ae308","Type":"ContainerStarted","Data":"5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc"} Nov 03 22:55:21.625230 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:55:21.625194 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:55:21.786302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:21.786263 5051 generic.go:356] "Generic (PLEG): container finished" podID="f66451bf-8657-4eb0-b592-b554812ae308" containerID="aac9334f820b70c31f2e778c8cadaf9ca91f41ffa7d52d0a87915ceb2e62922b" exitCode=0 Nov 03 22:55:21.786496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:21.786346 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" event={"ID":"f66451bf-8657-4eb0-b592-b554812ae308","Type":"ContainerDied","Data":"aac9334f820b70c31f2e778c8cadaf9ca91f41ffa7d52d0a87915ceb2e62922b"} Nov 03 22:55:22.993746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:22.993721 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:23.044628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.044600 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\") pod \"f66451bf-8657-4eb0-b592-b554812ae308\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " Nov 03 22:55:23.044792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.044717 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\") pod \"f66451bf-8657-4eb0-b592-b554812ae308\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " Nov 03 22:55:23.044792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.044772 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\" (UniqueName: \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\") pod \"f66451bf-8657-4eb0-b592-b554812ae308\" (UID: \"f66451bf-8657-4eb0-b592-b554812ae308\") " Nov 03 22:55:23.044953 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.044929 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca" pod "f66451bf-8657-4eb0-b592-b554812ae308" (UID: "f66451bf-8657-4eb0-b592-b554812ae308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:23.045419 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.045398 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:23.047062 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.047040 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569" pod "f66451bf-8657-4eb0-b592-b554812ae308" (UID: "f66451bf-8657-4eb0-b592-b554812ae308"). InnerVolumeSpecName "kube-api-access-zk569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:23.047147 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.047132 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca" pod "f66451bf-8657-4eb0-b592-b554812ae308" (UID: "f66451bf-8657-4eb0-b592-b554812ae308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:23.147803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.147758 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66451bf-8657-4eb0-b592-b554812ae308-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:23.147803 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.147797 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk569\" (UniqueName: \"kubernetes.io/projected/f66451bf-8657-4eb0-b592-b554812ae308-kube-api-access-zk569\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:23.813290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.813261 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" Nov 03 22:55:23.813290 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.813279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn" event={"ID":"f66451bf-8657-4eb0-b592-b554812ae308","Type":"ContainerDied","Data":"5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc"} Nov 03 22:55:23.813535 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:23.813339 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5363d32860ac3a67bf0f06fedb713f90b40f6f177a6efb5bd1827a0404f83ebc" Nov 03 22:55:30.784574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:30.784537 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:55:30.891250 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:30.891217 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:55:35.853241 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:35.796070 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" podUID="6cb94bb4-0312-4293-ae92-b433eb28f28d" containerName="payload-processing" containerID="cri-o://dd7f36507a5c2820bf96a9387e34d00dd641d245549a9baa8fe9c08a66cb6066" gracePeriod=25 Nov 03 22:55:35.894127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:35.894055 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" podUID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" containerName="payload-pre-processing" containerID="cri-o://9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e" gracePeriod=25 Nov 03 22:55:36.353642 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:55:36.353561 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6c9654_fe35_41d7_bffc_cfd9f4f3d3fd.slice/crio-conmon-9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:55:36.355743 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:55:36.353578 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6c9654_fe35_41d7_bffc_cfd9f4f3d3fd.slice/crio-conmon-9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:55:36.835771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:36.835729 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:55:36.836175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:36.836120 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:37.040823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.040737 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb"] Nov 03 22:55:37.042673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.042628 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f66451bf-8657-4eb0-b592-b554812ae308" containerName="revoke-keys" Nov 03 22:55:37.042878 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.042848 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66451bf-8657-4eb0-b592-b554812ae308" containerName="revoke-keys" Nov 03 22:55:37.043298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.043260 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66451bf-8657-4eb0-b592-b554812ae308" containerName="revoke-keys" Nov 03 22:55:37.339263 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.339228 5051 generic.go:356] "Generic (PLEG): container finished" podID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" containerID="9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e" exitCode=0 Nov 03 22:55:37.346013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.341898 5051 generic.go:356] "Generic (PLEG): container finished" podID="6cb94bb4-0312-4293-ae92-b433eb28f28d" containerID="dd7f36507a5c2820bf96a9387e34d00dd641d245549a9baa8fe9c08a66cb6066" exitCode=0 Nov 03 22:55:37.348731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.348699 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" event={"ID":"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd","Type":"ContainerDied","Data":"9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e"} Nov 03 22:55:37.348948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.348926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" event={"ID":"6cb94bb4-0312-4293-ae92-b433eb28f28d","Type":"ContainerDied","Data":"dd7f36507a5c2820bf96a9387e34d00dd641d245549a9baa8fe9c08a66cb6066"} Nov 03 22:55:37.349103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.349068 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.349215 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.349196 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb"] Nov 03 22:55:37.385332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.383394 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:55:37.416144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.416112 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:55:37.423593 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.417782 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\") pod \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " Nov 03 22:55:37.424321 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.424293 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\" (UniqueName: \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\") pod \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\" (UID: \"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd\") " Nov 03 22:55:37.424927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.424900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.425223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.425198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jq49\" (UniqueName: \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.425651 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.425624 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.425921 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.422233 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume" pod "3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" (UID: "3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:37.451482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.451441 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg" pod "3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" (UID: "3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd"). InnerVolumeSpecName "kube-api-access-drzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:37.472814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.472769 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:55:37.528151 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.528113 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jq49\" (UniqueName: \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.528564 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.528540 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.528788 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.528768 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.529033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.529016 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:37.529157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.529144 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drzqg\" (UniqueName: \"kubernetes.io/projected/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd-kube-api-access-drzqg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:37.530285 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.530262 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.533451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.532713 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.605043 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.604014 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jq49\" (UniqueName: \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\") pod \"maas-api-revoke-keys-e2e-api-a-26444f-r6kwb\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.635074 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.635047 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:55:37.731336 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.731304 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\" (UniqueName: \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\") pod \"6cb94bb4-0312-4293-ae92-b433eb28f28d\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " Nov 03 22:55:37.731584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.731565 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\") pod \"6cb94bb4-0312-4293-ae92-b433eb28f28d\" (UID: \"6cb94bb4-0312-4293-ae92-b433eb28f28d\") " Nov 03 22:55:37.731956 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.731923 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume" pod "6cb94bb4-0312-4293-ae92-b433eb28f28d" (UID: "6cb94bb4-0312-4293-ae92-b433eb28f28d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:37.732307 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.732280 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb94bb4-0312-4293-ae92-b433eb28f28d-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:37.733719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.733695 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq" pod "6cb94bb4-0312-4293-ae92-b433eb28f28d" (UID: "6cb94bb4-0312-4293-ae92-b433eb28f28d"). InnerVolumeSpecName "kube-api-access-pf9mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:37.760150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.760118 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:37.834131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:37.834083 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9mq\" (UniqueName: \"kubernetes.io/projected/6cb94bb4-0312-4293-ae92-b433eb28f28d-kube-api-access-pf9mq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:38.163917 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.163893 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb"] Nov 03 22:55:38.164641 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:55:38.164616 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca03d21_3462_4120_a1cc_363614cf0545.slice/crio-fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1 WatchSource:0}: Error finding container fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1: Status 404 returned error can't find the container with id fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1 Nov 03 22:55:38.357161 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.357129 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" event={"ID":"3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd","Type":"ContainerDied","Data":"0f272ae2fa1f33a56ef46301179ec987651c87f34f24a7d4889ec588dd67847d"} Nov 03 22:55:38.357354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.357188 5051 scope.go:122] "RemoveContainer" containerID="9e26fed7803f3f0e4e78046db420645da1d3fbcc77450b47022171576e47027e" Nov 03 22:55:38.357354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.357322 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs" Nov 03 22:55:38.373621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.363332 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" event={"ID":"6cb94bb4-0312-4293-ae92-b433eb28f28d","Type":"ContainerDied","Data":"ddf071dc6520690e34e1eec76e5393d98d0dc27486953e6a832508dfd97b482c"} Nov 03 22:55:38.373621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.363427 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f" Nov 03 22:55:38.373621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.369088 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" event={"ID":"cca03d21-3462-4120-a1cc-363614cf0545","Type":"ContainerStarted","Data":"fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1"} Nov 03 22:55:38.388629 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.388607 5051 scope.go:122] "RemoveContainer" containerID="dd7f36507a5c2820bf96a9387e34d00dd641d245549a9baa8fe9c08a66cb6066" Nov 03 22:55:38.527210 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.527134 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:55:38.576306 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.576281 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-efce83-84867f4d46-24l2f"] Nov 03 22:55:38.684607 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.684573 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:55:38.690132 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:38.690102 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-efce83-5c4b5c8f8c-xpbhs"] Nov 03 22:55:39.382573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:39.382541 5051 generic.go:356] "Generic (PLEG): container finished" podID="cca03d21-3462-4120-a1cc-363614cf0545" containerID="9e96c7f58095f1db3f81ec523e3f89f412aa355261643727179d5160c22de603" exitCode=0 Nov 03 22:55:39.383102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:39.382632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" event={"ID":"cca03d21-3462-4120-a1cc-363614cf0545","Type":"ContainerDied","Data":"9e96c7f58095f1db3f81ec523e3f89f412aa355261643727179d5160c22de603"} Nov 03 22:55:40.413510 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.413465 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" path="/var/lib/kubelet/pods/3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd/volumes" Nov 03 22:55:40.414183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.414160 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb94bb4-0312-4293-ae92-b433eb28f28d" path="/var/lib/kubelet/pods/6cb94bb4-0312-4293-ae92-b433eb28f28d/volumes" Nov 03 22:55:40.589542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.589515 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:40.680655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.680564 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\") pod \"cca03d21-3462-4120-a1cc-363614cf0545\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " Nov 03 22:55:40.680655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.680618 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\" (UniqueName: \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\") pod \"cca03d21-3462-4120-a1cc-363614cf0545\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " Nov 03 22:55:40.680862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.680765 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\") pod \"cca03d21-3462-4120-a1cc-363614cf0545\" (UID: \"cca03d21-3462-4120-a1cc-363614cf0545\") " Nov 03 22:55:40.681124 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.681092 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca" pod "cca03d21-3462-4120-a1cc-363614cf0545" (UID: "cca03d21-3462-4120-a1cc-363614cf0545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:40.681808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.681777 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:40.683206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.683176 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca" pod "cca03d21-3462-4120-a1cc-363614cf0545" (UID: "cca03d21-3462-4120-a1cc-363614cf0545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:55:40.683206 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.683191 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49" pod "cca03d21-3462-4120-a1cc-363614cf0545" (UID: "cca03d21-3462-4120-a1cc-363614cf0545"). InnerVolumeSpecName "kube-api-access-8jq49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:40.783367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.783326 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jq49\" (UniqueName: \"kubernetes.io/projected/cca03d21-3462-4120-a1cc-363614cf0545-kube-api-access-8jq49\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:40.783367 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:40.783359 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca03d21-3462-4120-a1cc-363614cf0545-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:41.401910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:41.401882 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" Nov 03 22:55:41.402165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:41.401880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb" event={"ID":"cca03d21-3462-4120-a1cc-363614cf0545","Type":"ContainerDied","Data":"fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1"} Nov 03 22:55:41.402165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:41.402039 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd54e3046e5d1cfc388b1ff6a1d6ed2c8deba433f083049ecea0dc43551b84c1" Nov 03 22:55:46.162818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:46.162710 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:55:47.012457 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.012429 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:55:47.013223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013206 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6cb94bb4-0312-4293-ae92-b433eb28f28d" containerName="payload-processing" Nov 03 22:55:47.013303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013224 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb94bb4-0312-4293-ae92-b433eb28f28d" containerName="payload-processing" Nov 03 22:55:47.013303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013236 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" containerName="payload-pre-processing" Nov 03 22:55:47.013303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013241 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" containerName="payload-pre-processing" Nov 03 22:55:47.013303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013256 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cca03d21-3462-4120-a1cc-363614cf0545" containerName="revoke-keys" Nov 03 22:55:47.013303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013261 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca03d21-3462-4120-a1cc-363614cf0545" containerName="revoke-keys" Nov 03 22:55:47.013478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013364 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb94bb4-0312-4293-ae92-b433eb28f28d" containerName="payload-processing" Nov 03 22:55:47.013478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013373 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6c9654-fe35-41d7-bffc-cfd9f4f3d3fd" containerName="payload-pre-processing" Nov 03 22:55:47.013478 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.013381 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca03d21-3462-4120-a1cc-363614cf0545" containerName="revoke-keys" Nov 03 22:55:47.023492 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.023475 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.030511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.030488 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:55:47.035271 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.035253 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.075607 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.075579 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-aacab9-openshift-default-dockercfg-p5sjp\"" Nov 03 22:55:47.122532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.122467 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:55:47.170871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.170643 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:55:47.210274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.210274 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.210527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210296 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.210527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210391 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210410 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210432 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210451 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210603 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.210750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210631 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210673 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdjq\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.210874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210741 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210874 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210848 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.210952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210903 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.211008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.210955 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpfl\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.211080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.211060 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.211134 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.211118 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.211218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.211200 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.211258 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.211240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.312801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312722 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.312801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312758 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.312801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312780 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312810 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312904 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.312938 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313019 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdjq\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313053 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313138 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313174 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpfl\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313219 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313222 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313295 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313319 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313354 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313385 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313432 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313480 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.313963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313583 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313608 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.313963 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313635 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.314883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.313935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.314883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.314534 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.315115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.315024 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.315226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.315197 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.317305 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.317276 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.317436 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.317335 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.318373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.318343 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.328212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.328137 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.332072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.332047 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.374877 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.374836 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.375079 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.375028 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.375223 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.375200 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpfl\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:47.375314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.375288 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdjq\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.642013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.641024 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:47.650900 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:47.650870 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:48.225573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.225408 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:55:48.229614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.229572 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.229729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.229663 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.229729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.229705 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.232354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.232328 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:55:48.277752 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:55:48.277724 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9108f4_a96b_413c_a8a8_9711644bb826.slice/crio-2c478e86c1e4cc49aa3733fc5247e53d348a68704737c7f5c3bcf144474d0633 WatchSource:0}: Error finding container 2c478e86c1e4cc49aa3733fc5247e53d348a68704737c7f5c3bcf144474d0633: Status 404 returned error can't find the container with id 2c478e86c1e4cc49aa3733fc5247e53d348a68704737c7f5c3bcf144474d0633 Nov 03 22:55:48.281221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.281180 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.281340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.281274 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.281340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.281316 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:48.465195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.464255 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" event={"ID":"c84c9481-15ed-4c60-9f7b-697adb1e2506","Type":"ContainerStarted","Data":"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069"} Nov 03 22:55:48.465195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.464309 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" event={"ID":"c84c9481-15ed-4c60-9f7b-697adb1e2506","Type":"ContainerStarted","Data":"786c13716e238b028419629aa20eebb982cfd3ff61d22fee4ecdfdd383432c02"} Nov 03 22:55:48.467072 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.467043 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" event={"ID":"ba9108f4-a96b-413c-a8a8-9711644bb826","Type":"ContainerStarted","Data":"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3"} Nov 03 22:55:48.467233 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.467088 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" event={"ID":"ba9108f4-a96b-413c-a8a8-9711644bb826","Type":"ContainerStarted","Data":"2c478e86c1e4cc49aa3733fc5247e53d348a68704737c7f5c3bcf144474d0633"} Nov 03 22:55:48.643537 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.643508 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:48.651938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.651914 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:48.710130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.709516 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podStartSLOduration=2.709498249 podStartE2EDuration="2.709498249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:48.7084254 +0000 UTC m=+2022.445769625" watchObservedRunningTime="2026-11-03 22:55:48.709498249 +0000 UTC m=+2022.446842452" Nov 03 22:55:48.712309 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:48.712261 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podStartSLOduration=2.712249999 podStartE2EDuration="2.712249999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:48.586754584 +0000 UTC m=+2022.324098830" watchObservedRunningTime="2026-11-03 22:55:48.712249999 +0000 UTC m=+2022.449594225" Nov 03 22:55:49.647011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:49.646247 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:55:49.647011 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:49.646368 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 22:55:49.655032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:49.654360 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:49.655032 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:49.654470 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:50.643381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:50.643350 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:55:50.643568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:50.643404 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 22:55:50.651725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:50.651699 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:50.652109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:50.651742 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:51.172696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.172620 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" podUID="17fdc5a3-17f3-425f-bc92-8aa731556312" containerName="payload-pre-processing" containerID="cri-o://971e960a4a897d7033328c03664a4df8366638549d885f1144c812f90e7f9fc7" gracePeriod=25 Nov 03 22:55:51.537686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.537650 5051 generic.go:356] "Generic (PLEG): container finished" podID="17fdc5a3-17f3-425f-bc92-8aa731556312" containerID="971e960a4a897d7033328c03664a4df8366638549d885f1144c812f90e7f9fc7" exitCode=0 Nov 03 22:55:51.538123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.538095 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" event={"ID":"17fdc5a3-17f3-425f-bc92-8aa731556312","Type":"ContainerDied","Data":"971e960a4a897d7033328c03664a4df8366638549d885f1144c812f90e7f9fc7"} Nov 03 22:55:51.650196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.649733 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:55:51.650196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.649792 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 22:55:51.652521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.652408 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:51.652521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.652457 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:51.720406 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:55:51.720288 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:55:51.729684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.729662 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:55:51.760027 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.756087 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\") pod \"17fdc5a3-17f3-425f-bc92-8aa731556312\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " Nov 03 22:55:51.765112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.765081 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\" (UniqueName: \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\") pod \"17fdc5a3-17f3-425f-bc92-8aa731556312\" (UID: \"17fdc5a3-17f3-425f-bc92-8aa731556312\") " Nov 03 22:55:51.800016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.787215 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume" pod "17fdc5a3-17f3-425f-bc92-8aa731556312" (UID: "17fdc5a3-17f3-425f-bc92-8aa731556312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:51.800016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.795169 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk" pod "17fdc5a3-17f3-425f-bc92-8aa731556312" (UID: "17fdc5a3-17f3-425f-bc92-8aa731556312"). InnerVolumeSpecName "kube-api-access-glltk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:51.919020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.918321 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fdc5a3-17f3-425f-bc92-8aa731556312-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:51.919020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:51.918357 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glltk\" (UniqueName: \"kubernetes.io/projected/17fdc5a3-17f3-425f-bc92-8aa731556312-kube-api-access-glltk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:52.548569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.548539 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" Nov 03 22:55:52.548720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.548677 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq" event={"ID":"17fdc5a3-17f3-425f-bc92-8aa731556312","Type":"ContainerDied","Data":"b930da756d221aac8e7884d551fd9d66d342cf0172426ae41950783b435655e7"} Nov 03 22:55:52.548780 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.548725 5051 scope.go:122] "RemoveContainer" containerID="971e960a4a897d7033328c03664a4df8366638549d885f1144c812f90e7f9fc7" Nov 03 22:55:52.643948 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.643914 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:55:52.644383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.643968 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 22:55:52.651681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.651655 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:52.651800 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.651722 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:52.725766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.725660 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:55:52.730386 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:52.730360 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-26444f-67b5fb89f-jsmbq"] Nov 03 22:55:53.643524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:53.643481 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 22:55:53.643742 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:53.643537 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 22:55:53.651303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:53.651273 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:53.651454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:53.651320 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:54.418725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.418689 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fdc5a3-17f3-425f-bc92-8aa731556312" path="/var/lib/kubelet/pods/17fdc5a3-17f3-425f-bc92-8aa731556312/volumes" Nov 03 22:55:54.652351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.652313 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 22:55:54.652540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.652370 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 22:55:54.653255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.653123 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:54.653623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.653523 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:54.654820 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:54.654792 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:55:55.089249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.089218 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:55:55.089438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.089418 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" containerID="cri-o://933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4" gracePeriod=30 Nov 03 22:55:55.500711 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.500668 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:55:55.501240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.500912 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" containerID="cri-o://1779609fc8157c3d31cd0f2208fcad51de087678ced13de42254c5db3a7a3e46" gracePeriod=30 Nov 03 22:55:55.656494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.656462 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:55.656970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.656950 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:55.657617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.657596 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:55:55.773084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.772958 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:55:55.773316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:55.773208 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" containerID="cri-o://7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c" gracePeriod=30 Nov 03 22:55:56.870898 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:56.870850 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:56.871524 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:56.870917 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:59.105734 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:59.105697 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:59.106116 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:55:59.105763 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:56:00.402589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.402560 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:56:00.575660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575606 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.575866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575681 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.575866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575733 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.575866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575771 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.575866 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575837 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.576150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575885 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.576150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.575940 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.576150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.576090 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.576150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.576122 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\") pod \"8de53ca3-628d-492a-bc0f-69387afd86d2\" (UID: \"8de53ca3-628d-492a-bc0f-69387afd86d2\") " Nov 03 22:56:00.577178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.576854 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.577178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.576901 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.577707 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.577414 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.582751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.582724 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.587368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.587111 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:00.587368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.587247 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:00.587914 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.587720 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.588194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.588079 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.588356 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.588339 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.589933 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.588474 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.610626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.610529 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "kube-api-access-2pdl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:00.612287 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.612200 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:00.616183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.616147 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy" pod "8de53ca3-628d-492a-bc0f-69387afd86d2" (UID: "8de53ca3-628d-492a-bc0f-69387afd86d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.685893 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.685850 5051 generic.go:356] "Generic (PLEG): container finished" podID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerID="933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4" exitCode=0 Nov 03 22:56:00.686129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.685956 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" event={"ID":"8de53ca3-628d-492a-bc0f-69387afd86d2","Type":"ContainerDied","Data":"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4"} Nov 03 22:56:00.686129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.686059 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" event={"ID":"8de53ca3-628d-492a-bc0f-69387afd86d2","Type":"ContainerDied","Data":"d8b5fff26a1944d2b6ebbdb95e5441d2b0456bfc91ba2bb85fb9d80a9439d22f"} Nov 03 22:56:00.686129 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.686090 5051 scope.go:122] "RemoveContainer" containerID="933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4" Nov 03 22:56:00.686323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.686263 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96" Nov 03 22:56:00.691951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.691562 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.691951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.691590 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdl4\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-kube-api-access-2pdl4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.691951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.691605 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de53ca3-628d-492a-bc0f-69387afd86d2-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.691951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.691618 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.691951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.691632 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de53ca3-628d-492a-bc0f-69387afd86d2-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.693033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.692860 5051 generic.go:356] "Generic (PLEG): container finished" podID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerID="1779609fc8157c3d31cd0f2208fcad51de087678ced13de42254c5db3a7a3e46" exitCode=0 Nov 03 22:56:00.693033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.692924 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" event={"ID":"9ff874b2-62ba-4c52-9183-843cfd37a3d0","Type":"ContainerDied","Data":"1779609fc8157c3d31cd0f2208fcad51de087678ced13de42254c5db3a7a3e46"} Nov 03 22:56:00.714100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.714071 5051 scope.go:122] "RemoveContainer" containerID="933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4" Nov 03 22:56:00.714473 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:00.714449 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4\": container with ID starting with 933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4 not found: ID does not exist" containerID="933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4" Nov 03 22:56:00.714586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.714482 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4"} err="failed to get container status \"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4\": rpc error: code = NotFound desc = could not find container \"933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4\": container with ID starting with 933bbc67b62d601a788d0a7c78ba2e9b70fe9596e915a81b3c57616930874dc4 not found: ID does not exist" Nov 03 22:56:00.856193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.856163 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:56:00.860784 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.860756 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-hsn96"] Nov 03 22:56:00.890211 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.890185 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997474 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997632 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997701 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997790 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997849 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997904 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.997945 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.998059 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:00.999338 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:00.998100 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\") pod \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\" (UID: \"9ff874b2-62ba-4c52-9183-843cfd37a3d0\") " Nov 03 22:56:01.003491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.003182 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.003976 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.003768 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.004425 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.004162 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.006508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.005907 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "kube-api-access-qglc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.006508 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.006260 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:01.008128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.008090 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.008128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.008111 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.009157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.009125 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:01.017430 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.017400 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy" pod "9ff874b2-62ba-4c52-9183-843cfd37a3d0" (UID: "9ff874b2-62ba-4c52-9183-843cfd37a3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.090181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.090156 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:56:01.100142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100117 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100142 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100143 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100159 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglc9\" (UniqueName: \"kubernetes.io/projected/9ff874b2-62ba-4c52-9183-843cfd37a3d0-kube-api-access-qglc9\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100171 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100182 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100193 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100206 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100218 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.100324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.100230 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff874b2-62ba-4c52-9183-843cfd37a3d0-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.201339 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201303 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201361 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201462 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201526 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201572 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201677 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.201841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.201821 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.202045 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202015 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.202136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202078 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.202194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202165 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.202249 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202231 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.202377 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202354 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.202507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.202431 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\") pod \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\" (UID: \"0e4eeffa-f6fd-47e7-a126-d55d80948ace\") " Nov 03 22:56:01.203228 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.203207 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.203375 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.203361 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.203504 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.203490 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.203616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.203605 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.203729 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.203257 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:01.204778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.204743 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:01.205966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.205941 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.206288 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.206255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.206869 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.206846 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9" pod "0e4eeffa-f6fd-47e7-a126-d55d80948ace" (UID: "0e4eeffa-f6fd-47e7-a126-d55d80948ace"). InnerVolumeSpecName "kube-api-access-wjrz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.305746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.305705 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.305746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.305743 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.305746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.305753 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.305984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.305766 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjrz9\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-kube-api-access-wjrz9\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.305984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.305775 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eeffa-f6fd-47e7-a126-d55d80948ace-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.706685 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.706657 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" Nov 03 22:56:01.707165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.706687 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2" event={"ID":"9ff874b2-62ba-4c52-9183-843cfd37a3d0","Type":"ContainerDied","Data":"e73f0b5eb0a2a490b7f2af2b4689d68a56d9f0beb185f44da1415b789b6dff56"} Nov 03 22:56:01.707165 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.706747 5051 scope.go:122] "RemoveContainer" containerID="1779609fc8157c3d31cd0f2208fcad51de087678ced13de42254c5db3a7a3e46" Nov 03 22:56:01.717471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.716953 5051 generic.go:356] "Generic (PLEG): container finished" podID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerID="7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c" exitCode=0 Nov 03 22:56:01.717471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.717131 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" event={"ID":"0e4eeffa-f6fd-47e7-a126-d55d80948ace","Type":"ContainerDied","Data":"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c"} Nov 03 22:56:01.717471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.717206 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" event={"ID":"0e4eeffa-f6fd-47e7-a126-d55d80948ace","Type":"ContainerDied","Data":"534ce9a44ea4b18db18792737a4eeb70ac180d7ff770a362402f048359237acd"} Nov 03 22:56:01.717471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.717315 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh" Nov 03 22:56:01.732053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.732032 5051 scope.go:122] "RemoveContainer" containerID="7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c" Nov 03 22:56:01.743616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.743593 5051 scope.go:122] "RemoveContainer" containerID="7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c" Nov 03 22:56:01.743978 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:01.743948 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c\": container with ID starting with 7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c not found: ID does not exist" containerID="7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c" Nov 03 22:56:01.744085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.744024 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c"} err="failed to get container status \"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c\": rpc error: code = NotFound desc = could not find container \"7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c\": container with ID starting with 7e7ca8cc6d78792c7aa6f89fc15d4f684eb4ce733ef3cd2eb98ae5d6e5d4e00c not found: ID does not exist" Nov 03 22:56:01.884067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.884032 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:56:01.892178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:01.892146 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-klsq2"] Nov 03 22:56:02.008184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:02.008094 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:56:02.011480 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:02.011460 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-5tnzh"] Nov 03 22:56:02.410252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:02.410219 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" path="/var/lib/kubelet/pods/0e4eeffa-f6fd-47e7-a126-d55d80948ace/volumes" Nov 03 22:56:02.410757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:02.410743 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" path="/var/lib/kubelet/pods/8de53ca3-628d-492a-bc0f-69387afd86d2/volumes" Nov 03 22:56:02.411214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:02.411200 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" path="/var/lib/kubelet/pods/9ff874b2-62ba-4c52-9183-843cfd37a3d0/volumes" Nov 03 22:56:03.108007 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.107962 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk"] Nov 03 22:56:03.108907 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.108883 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.108908 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.108930 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.108937 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.108978 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109005 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" Nov 03 22:56:03.109046 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109040 5051 cpu_manager.go:414] "containerMap: removing container" podUID="17fdc5a3-17f3-425f-bc92-8aa731556312" containerName="payload-pre-processing" Nov 03 22:56:03.109366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109048 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fdc5a3-17f3-425f-bc92-8aa731556312" containerName="payload-pre-processing" Nov 03 22:56:03.109366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109259 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de53ca3-628d-492a-bc0f-69387afd86d2" containerName="istio-proxy" Nov 03 22:56:03.109366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109277 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4eeffa-f6fd-47e7-a126-d55d80948ace" containerName="istio-proxy" Nov 03 22:56:03.109366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109293 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff874b2-62ba-4c52-9183-843cfd37a3d0" containerName="istio-proxy" Nov 03 22:56:03.109366 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.109309 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fdc5a3-17f3-425f-bc92-8aa731556312" containerName="payload-pre-processing" Nov 03 22:56:03.114642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.114622 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.137488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.137451 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62njk\" (UniqueName: \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.137644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.137517 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.137644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.137611 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.169694 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.169664 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk"] Nov 03 22:56:03.212637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.212603 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:03.212637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.212619 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:56:03.239693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.239662 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62njk\" (UniqueName: \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.239870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.239718 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.239870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.239739 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.240604 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.240573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.242774 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.242752 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.302549 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.302523 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62njk\" (UniqueName: \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.429598 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.429518 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:03.822709 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:03.822672 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk"] Nov 03 22:56:03.824066 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:56:03.824033 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79629b14_d38c_41e3_8892_7301e4aa5a4b.slice/crio-3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12 WatchSource:0}: Error finding container 3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12: Status 404 returned error can't find the container with id 3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12 Nov 03 22:56:04.746101 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:04.746061 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" event={"ID":"79629b14-d38c-41e3-8892-7301e4aa5a4b","Type":"ContainerStarted","Data":"8f9d2ae36c709a84dc231366c8bcd6e486e99982b703b7a0b4e0685845c3548f"} Nov 03 22:56:04.746494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:04.746123 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" event={"ID":"79629b14-d38c-41e3-8892-7301e4aa5a4b","Type":"ContainerStarted","Data":"3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12"} Nov 03 22:56:04.889889 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:04.889830 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" podStartSLOduration=1.889816503 podStartE2EDuration="1.889816503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:04.888415411 +0000 UTC m=+2038.625759636" watchObservedRunningTime="2026-11-03 22:56:04.889816503 +0000 UTC m=+2038.627160728" Nov 03 22:56:05.755459 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:05.755425 5051 generic.go:356] "Generic (PLEG): container finished" podID="79629b14-d38c-41e3-8892-7301e4aa5a4b" containerID="8f9d2ae36c709a84dc231366c8bcd6e486e99982b703b7a0b4e0685845c3548f" exitCode=0 Nov 03 22:56:05.755872 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:05.755483 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" event={"ID":"79629b14-d38c-41e3-8892-7301e4aa5a4b","Type":"ContainerDied","Data":"8f9d2ae36c709a84dc231366c8bcd6e486e99982b703b7a0b4e0685845c3548f"} Nov 03 22:56:06.935628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.935604 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:06.982083 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.982052 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\") pod \"79629b14-d38c-41e3-8892-7301e4aa5a4b\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " Nov 03 22:56:06.982252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.982187 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\") pod \"79629b14-d38c-41e3-8892-7301e4aa5a4b\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " Nov 03 22:56:06.982318 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.982290 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\" (UniqueName: \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\") pod \"79629b14-d38c-41e3-8892-7301e4aa5a4b\" (UID: \"79629b14-d38c-41e3-8892-7301e4aa5a4b\") " Nov 03 22:56:06.982598 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.982575 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca" pod "79629b14-d38c-41e3-8892-7301e4aa5a4b" (UID: "79629b14-d38c-41e3-8892-7301e4aa5a4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:06.983041 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.982978 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:06.984519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.984494 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca" pod "79629b14-d38c-41e3-8892-7301e4aa5a4b" (UID: "79629b14-d38c-41e3-8892-7301e4aa5a4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:06.985096 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:06.985072 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk" pod "79629b14-d38c-41e3-8892-7301e4aa5a4b" (UID: "79629b14-d38c-41e3-8892-7301e4aa5a4b"). InnerVolumeSpecName "kube-api-access-62njk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:07.084969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:07.084867 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62njk\" (UniqueName: \"kubernetes.io/projected/79629b14-d38c-41e3-8892-7301e4aa5a4b-kube-api-access-62njk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:07.084969 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:07.084899 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79629b14-d38c-41e3-8892-7301e4aa5a4b-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:07.773642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:07.773611 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" Nov 03 22:56:07.773642 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:07.773629 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk" event={"ID":"79629b14-d38c-41e3-8892-7301e4aa5a4b","Type":"ContainerDied","Data":"3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12"} Nov 03 22:56:07.773875 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:07.773675 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6b833c8fa1b8ae83fd1d60b0a7729f024f2a9cb0a3b6760de342941921dd12" Nov 03 22:56:09.289558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:09.289519 5051 scope.go:122] "RemoveContainer" containerID="2ade9c03dcebb86b1ff90409cafe5911dc7959ede2a2f3cc62686ac36cc93c74" Nov 03 22:56:09.300314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:09.300295 5051 scope.go:122] "RemoveContainer" containerID="3ab6e1e8881670a24d1f620812a43d14f2507014ffc3a4a71636b7e4c2f6bef5" Nov 03 22:56:14.593244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:14.593212 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:56:14.710042 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:14.709959 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:56:19.594244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.594189 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" podUID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" containerName="payload-processing" containerID="cri-o://246965c6eb4cb3649970b01d22798e4bdbf19f99a769c019c4c8879d681c9e02" gracePeriod=25 Nov 03 22:56:19.710715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.710664 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" podUID="3b33f9c7-1553-4fad-8ea0-863239729460" containerName="payload-pre-processing" containerID="cri-o://76aa92bb5aa3a98ae0fc74b612a2f827d48483fdb7594800bd5104c319573cc7" gracePeriod=25 Nov 03 22:56:19.885061 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.884977 5051 generic.go:356] "Generic (PLEG): container finished" podID="3b33f9c7-1553-4fad-8ea0-863239729460" containerID="76aa92bb5aa3a98ae0fc74b612a2f827d48483fdb7594800bd5104c319573cc7" exitCode=0 Nov 03 22:56:19.885210 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.885068 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" event={"ID":"3b33f9c7-1553-4fad-8ea0-863239729460","Type":"ContainerDied","Data":"76aa92bb5aa3a98ae0fc74b612a2f827d48483fdb7594800bd5104c319573cc7"} Nov 03 22:56:19.888439 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.888413 5051 generic.go:356] "Generic (PLEG): container finished" podID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" containerID="246965c6eb4cb3649970b01d22798e4bdbf19f99a769c019c4c8879d681c9e02" exitCode=0 Nov 03 22:56:19.888589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.888487 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" event={"ID":"53254326-f0dc-42bf-8edc-e1d7ccb656ef","Type":"ContainerDied","Data":"246965c6eb4cb3649970b01d22798e4bdbf19f99a769c019c4c8879d681c9e02"} Nov 03 22:56:19.888589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.888549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" event={"ID":"53254326-f0dc-42bf-8edc-e1d7ccb656ef","Type":"ContainerDied","Data":"f4ab43aa3fde6ddb6c0db6e41a25c3b6a89bb5a9c26b39e3d226ef7e1cdf5baf"} Nov 03 22:56:19.888589 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.888574 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ab43aa3fde6ddb6c0db6e41a25c3b6a89bb5a9c26b39e3d226ef7e1cdf5baf" Nov 03 22:56:19.891902 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.891885 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:56:19.984871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.984840 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\") pod \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " Nov 03 22:56:19.985254 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.985231 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\" (UniqueName: \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\") pod \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\" (UID: \"53254326-f0dc-42bf-8edc-e1d7ccb656ef\") " Nov 03 22:56:19.985380 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.985352 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume" pod "53254326-f0dc-42bf-8edc-e1d7ccb656ef" (UID: "53254326-f0dc-42bf-8edc-e1d7ccb656ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:19.985885 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.985863 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53254326-f0dc-42bf-8edc-e1d7ccb656ef-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:19.988406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:19.988373 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5" pod "53254326-f0dc-42bf-8edc-e1d7ccb656ef" (UID: "53254326-f0dc-42bf-8edc-e1d7ccb656ef"). InnerVolumeSpecName "kube-api-access-wlhw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:20.014741 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.014717 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:56:20.087219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.087186 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\" (UniqueName: \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\") pod \"3b33f9c7-1553-4fad-8ea0-863239729460\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " Nov 03 22:56:20.087402 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.087243 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\") pod \"3b33f9c7-1553-4fad-8ea0-863239729460\" (UID: \"3b33f9c7-1553-4fad-8ea0-863239729460\") " Nov 03 22:56:20.087691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.087666 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume" pod "3b33f9c7-1553-4fad-8ea0-863239729460" (UID: "3b33f9c7-1553-4fad-8ea0-863239729460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:20.088003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.087956 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b33f9c7-1553-4fad-8ea0-863239729460-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:20.088003 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.087978 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhw5\" (UniqueName: \"kubernetes.io/projected/53254326-f0dc-42bf-8edc-e1d7ccb656ef-kube-api-access-wlhw5\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:20.089696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.089663 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx" pod "3b33f9c7-1553-4fad-8ea0-863239729460" (UID: "3b33f9c7-1553-4fad-8ea0-863239729460"). InnerVolumeSpecName "kube-api-access-7n7mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:20.188973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.188882 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7mx\" (UniqueName: \"kubernetes.io/projected/3b33f9c7-1553-4fad-8ea0-863239729460-kube-api-access-7n7mx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:20.897484 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.897398 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" Nov 03 22:56:20.897484 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.897427 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw" event={"ID":"3b33f9c7-1553-4fad-8ea0-863239729460","Type":"ContainerDied","Data":"9d742f6917130b9cb715f0b85c6a61e3fa4c38c3b2de038ebec13b9774e48205"} Nov 03 22:56:20.898037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.897499 5051 scope.go:122] "RemoveContainer" containerID="76aa92bb5aa3a98ae0fc74b612a2f827d48483fdb7594800bd5104c319573cc7" Nov 03 22:56:20.898037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:20.897591 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp" Nov 03 22:56:21.117047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:21.117015 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:56:21.121139 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:21.121110 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ea1be0-74784f4fdd-5g9xp"] Nov 03 22:56:21.276255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:21.276175 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:56:21.281064 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:21.281042 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ea1be0-59585bc9f9-766gw"] Nov 03 22:56:21.625086 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:21.625049 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:56:22.411588 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:22.411559 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b33f9c7-1553-4fad-8ea0-863239729460" path="/var/lib/kubelet/pods/3b33f9c7-1553-4fad-8ea0-863239729460/volumes" Nov 03 22:56:22.412114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:22.412013 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" path="/var/lib/kubelet/pods/53254326-f0dc-42bf-8edc-e1d7ccb656ef/volumes" Nov 03 22:56:24.192703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.192667 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx"] Nov 03 22:56:24.193460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193444 5051 cpu_manager.go:414] "containerMap: removing container" podUID="79629b14-d38c-41e3-8892-7301e4aa5a4b" containerName="revoke-keys" Nov 03 22:56:24.193532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193462 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79629b14-d38c-41e3-8892-7301e4aa5a4b" containerName="revoke-keys" Nov 03 22:56:24.193532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193486 5051 cpu_manager.go:414] "containerMap: removing container" podUID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" containerName="payload-processing" Nov 03 22:56:24.193532 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193491 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" containerName="payload-processing" Nov 03 22:56:24.193649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193540 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3b33f9c7-1553-4fad-8ea0-863239729460" containerName="payload-pre-processing" Nov 03 22:56:24.193649 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193545 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b33f9c7-1553-4fad-8ea0-863239729460" containerName="payload-pre-processing" Nov 03 22:56:24.193715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193652 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b33f9c7-1553-4fad-8ea0-863239729460" containerName="payload-pre-processing" Nov 03 22:56:24.193715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193660 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="79629b14-d38c-41e3-8892-7301e4aa5a4b" containerName="revoke-keys" Nov 03 22:56:24.193715 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.193669 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="53254326-f0dc-42bf-8edc-e1d7ccb656ef" containerName="payload-processing" Nov 03 22:56:24.197533 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.197511 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.244913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.244882 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rs74\" (UniqueName: \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.245135 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.244948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.245186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.245137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.252103 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.252075 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:56:24.258009 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.257969 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx"] Nov 03 22:56:24.300102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.300073 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:56:24.346771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.346742 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.346944 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.346837 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.347138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.347116 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rs74\" (UniqueName: \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.347568 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.347544 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.349239 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.349212 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.408802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.408763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rs74\" (UniqueName: \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.512507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.512424 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:24.899860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.896962 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx"] Nov 03 22:56:24.930343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:24.930314 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" event={"ID":"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8","Type":"ContainerStarted","Data":"1f217b9117214c1f8104e4c18d86248ee2ad4fca8ea6d48816a85be8672fe5a6"} Nov 03 22:56:25.941683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:25.941653 5051 generic.go:356] "Generic (PLEG): container finished" podID="a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" containerID="311940f5a2a9e38437230e2bc6f43e177df4f9608b82976f6b6191ad8a051fd1" exitCode=0 Nov 03 22:56:25.942090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:25.941703 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" event={"ID":"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8","Type":"ContainerDied","Data":"311940f5a2a9e38437230e2bc6f43e177df4f9608b82976f6b6191ad8a051fd1"} Nov 03 22:56:26.283077 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.282980 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7"] Nov 03 22:56:26.287586 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.287569 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.345513 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.345483 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7"] Nov 03 22:56:26.382471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.382438 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.382652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.382501 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgqj\" (UniqueName: \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.382652 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.382624 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.483845 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.483813 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.484056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.483858 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgqj\" (UniqueName: \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.484056 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.483912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.484664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.484643 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.486625 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.486607 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.553605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.553519 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgqj\" (UniqueName: \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\") pod \"maas-api-revoke-sub-6682c402-48fw7\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.603049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.603014 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:26.995031 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:26.994899 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7"] Nov 03 22:56:26.996025 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:56:26.995975 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b427274_491f_4a08_9e61_406e24df1797.slice/crio-79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5 WatchSource:0}: Error finding container 79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5: Status 404 returned error can't find the container with id 79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5 Nov 03 22:56:27.113479 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.113452 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:27.195654 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.195621 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\" (UniqueName: \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\") pod \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " Nov 03 22:56:27.195851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.195837 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\") pod \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " Nov 03 22:56:27.195935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.195923 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\") pod \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\" (UID: \"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8\") " Nov 03 22:56:27.196437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.196344 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca" pod "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" (UID: "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:27.196705 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.196682 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:27.198420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.198391 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74" pod "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" (UID: "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8"). InnerVolumeSpecName "kube-api-access-4rs74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:27.198509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.198473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca" pod "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" (UID: "a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:27.298060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.297949 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rs74\" (UniqueName: \"kubernetes.io/projected/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-kube-api-access-4rs74\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:27.298060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.297980 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:27.311444 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.311412 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5"] Nov 03 22:56:27.312184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.312165 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" containerName="revoke-keys" Nov 03 22:56:27.312184 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.312181 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" containerName="revoke-keys" Nov 03 22:56:27.312376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.312307 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" containerName="revoke-keys" Nov 03 22:56:27.316122 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.316102 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.376332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.375751 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5"] Nov 03 22:56:27.399582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.399231 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.399582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.399385 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhx6v\" (UniqueName: \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.399582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.399421 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.500799 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.500744 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.500961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.500894 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhx6v\" (UniqueName: \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.500961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.500932 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.501582 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.501559 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.503938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.503912 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.562025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.561936 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhx6v\" (UniqueName: \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\") pod \"maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.636176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.636143 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:27.960908 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.960871 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" event={"ID":"5b427274-491f-4a08-9e61-406e24df1797","Type":"ContainerStarted","Data":"cae2a85df2b553630a858c06d1814d34d03352b674d7583595e2f728ea7d3b32"} Nov 03 22:56:27.961141 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.960931 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" event={"ID":"5b427274-491f-4a08-9e61-406e24df1797","Type":"ContainerStarted","Data":"79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5"} Nov 03 22:56:27.963209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.963178 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" Nov 03 22:56:27.963209 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.963196 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx" event={"ID":"a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8","Type":"ContainerDied","Data":"1f217b9117214c1f8104e4c18d86248ee2ad4fca8ea6d48816a85be8672fe5a6"} Nov 03 22:56:27.963405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:27.963233 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f217b9117214c1f8104e4c18d86248ee2ad4fca8ea6d48816a85be8672fe5a6" Nov 03 22:56:28.031810 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.031787 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5"] Nov 03 22:56:28.032141 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:56:28.031870 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857d556c_76e0_424a_b0dc_4fa1e7f1066c.slice/crio-6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a WatchSource:0}: Error finding container 6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a: Status 404 returned error can't find the container with id 6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a Nov 03 22:56:28.078966 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.078919 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" podStartSLOduration=2.07890516 podStartE2EDuration="2.07890516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:28.076714278 +0000 UTC m=+2061.814058514" watchObservedRunningTime="2026-11-03 22:56:28.07890516 +0000 UTC m=+2061.816249426" Nov 03 22:56:28.973523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.973488 5051 generic.go:356] "Generic (PLEG): container finished" podID="5b427274-491f-4a08-9e61-406e24df1797" containerID="cae2a85df2b553630a858c06d1814d34d03352b674d7583595e2f728ea7d3b32" exitCode=0 Nov 03 22:56:28.973747 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.973574 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" event={"ID":"5b427274-491f-4a08-9e61-406e24df1797","Type":"ContainerDied","Data":"cae2a85df2b553630a858c06d1814d34d03352b674d7583595e2f728ea7d3b32"} Nov 03 22:56:28.975511 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.975485 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" event={"ID":"857d556c-76e0-424a-b0dc-4fa1e7f1066c","Type":"ContainerStarted","Data":"6a1a5173993b0bf7d3b7cca35865771e890249e79d3c0c6c02f7ceca5cb62b47"} Nov 03 22:56:28.975639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:28.975530 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" event={"ID":"857d556c-76e0-424a-b0dc-4fa1e7f1066c","Type":"ContainerStarted","Data":"6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a"} Nov 03 22:56:29.205377 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:29.205319 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" podStartSLOduration=2.20530416 podStartE2EDuration="2.20530416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:56:29.202823683 +0000 UTC m=+2062.940167927" watchObservedRunningTime="2026-11-03 22:56:29.20530416 +0000 UTC m=+2062.942648388" Nov 03 22:56:29.985697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:29.985661 5051 generic.go:356] "Generic (PLEG): container finished" podID="857d556c-76e0-424a-b0dc-4fa1e7f1066c" containerID="6a1a5173993b0bf7d3b7cca35865771e890249e79d3c0c6c02f7ceca5cb62b47" exitCode=0 Nov 03 22:56:29.985927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:29.985746 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" event={"ID":"857d556c-76e0-424a-b0dc-4fa1e7f1066c","Type":"ContainerDied","Data":"6a1a5173993b0bf7d3b7cca35865771e890249e79d3c0c6c02f7ceca5cb62b47"} Nov 03 22:56:30.192961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.192929 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:30.347354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.347265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\") pod \"5b427274-491f-4a08-9e61-406e24df1797\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " Nov 03 22:56:30.347844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.347443 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\") pod \"5b427274-491f-4a08-9e61-406e24df1797\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " Nov 03 22:56:30.347844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.347494 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\" (UniqueName: \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\") pod \"5b427274-491f-4a08-9e61-406e24df1797\" (UID: \"5b427274-491f-4a08-9e61-406e24df1797\") " Nov 03 22:56:30.349020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.347851 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca" pod "5b427274-491f-4a08-9e61-406e24df1797" (UID: "5b427274-491f-4a08-9e61-406e24df1797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:30.349020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.348314 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b427274-491f-4a08-9e61-406e24df1797-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.350078 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.350045 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca" pod "5b427274-491f-4a08-9e61-406e24df1797" (UID: "5b427274-491f-4a08-9e61-406e24df1797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:30.350624 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.350595 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj" pod "5b427274-491f-4a08-9e61-406e24df1797" (UID: "5b427274-491f-4a08-9e61-406e24df1797"). InnerVolumeSpecName "kube-api-access-6qgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:30.449841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.449806 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b427274-491f-4a08-9e61-406e24df1797-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:30.449841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:30.449836 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgqj\" (UniqueName: \"kubernetes.io/projected/5b427274-491f-4a08-9e61-406e24df1797-kube-api-access-6qgqj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:31.000177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.000147 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" Nov 03 22:56:31.000387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.000172 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7" event={"ID":"5b427274-491f-4a08-9e61-406e24df1797","Type":"ContainerDied","Data":"79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5"} Nov 03 22:56:31.000387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.000288 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c95f13dbcb52a9d42d5fb0c392a360f3a4eadc1f49f02411c53c71850e55e5" Nov 03 22:56:31.200622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.200594 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:31.365301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.365244 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\") pod \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " Nov 03 22:56:31.365824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.365315 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\") pod \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " Nov 03 22:56:31.365824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.365704 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca" pod "857d556c-76e0-424a-b0dc-4fa1e7f1066c" (UID: "857d556c-76e0-424a-b0dc-4fa1e7f1066c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:31.365824 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.365717 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\" (UniqueName: \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\") pod \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\" (UID: \"857d556c-76e0-424a-b0dc-4fa1e7f1066c\") " Nov 03 22:56:31.367050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.367021 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:31.368552 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.368521 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca" pod "857d556c-76e0-424a-b0dc-4fa1e7f1066c" (UID: "857d556c-76e0-424a-b0dc-4fa1e7f1066c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:31.368766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.368734 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v" pod "857d556c-76e0-424a-b0dc-4fa1e7f1066c" (UID: "857d556c-76e0-424a-b0dc-4fa1e7f1066c"). InnerVolumeSpecName "kube-api-access-mhx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:31.468781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.468724 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857d556c-76e0-424a-b0dc-4fa1e7f1066c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:31.468781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:31.468770 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhx6v\" (UniqueName: \"kubernetes.io/projected/857d556c-76e0-424a-b0dc-4fa1e7f1066c-kube-api-access-mhx6v\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:32.010722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:32.010696 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" Nov 03 22:56:32.010920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:32.010739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5" event={"ID":"857d556c-76e0-424a-b0dc-4fa1e7f1066c","Type":"ContainerDied","Data":"6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a"} Nov 03 22:56:32.010920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:32.010785 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f4d39b78de849f617e6d15f2b4e73b4a8dc44181eac6096974bc520a01b783a" Nov 03 22:56:38.033507 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:38.033470 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:56:38.148090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:38.148052 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:56:40.600580 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:40.600546 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:56:40.719149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:40.719107 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:56:43.034616 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.034578 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" podUID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" containerName="payload-processing" containerID="cri-o://fc6ff0a921593e13ddeb5d83e1239fdae4b7bee7a7ea86e3f4b17797a90ebb51" gracePeriod=25 Nov 03 22:56:43.154370 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.154285 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" podUID="081e4e07-8050-499a-8f1f-0771a825bf4d" containerName="payload-pre-processing" containerID="cri-o://c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f" gracePeriod=25 Nov 03 22:56:43.156137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.156108 5051 generic.go:356] "Generic (PLEG): container finished" podID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" containerID="fc6ff0a921593e13ddeb5d83e1239fdae4b7bee7a7ea86e3f4b17797a90ebb51" exitCode=0 Nov 03 22:56:43.156454 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.156430 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" event={"ID":"ca99ec5e-7270-4561-9e62-15b3950eb8c9","Type":"ContainerDied","Data":"fc6ff0a921593e13ddeb5d83e1239fdae4b7bee7a7ea86e3f4b17797a90ebb51"} Nov 03 22:56:43.246816 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:43.246776 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081e4e07_8050_499a_8f1f_0771a825bf4d.slice/crio-conmon-c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:56:43.355067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.355037 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:56:43.428278 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.428243 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\") pod \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " Nov 03 22:56:43.428458 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.428319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\" (UniqueName: \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\") pod \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\" (UID: \"ca99ec5e-7270-4561-9e62-15b3950eb8c9\") " Nov 03 22:56:43.428660 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.428634 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume" pod "ca99ec5e-7270-4561-9e62-15b3950eb8c9" (UID: "ca99ec5e-7270-4561-9e62-15b3950eb8c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:43.429040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.428942 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99ec5e-7270-4561-9e62-15b3950eb8c9-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:43.430925 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.430893 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p" pod "ca99ec5e-7270-4561-9e62-15b3950eb8c9" (UID: "ca99ec5e-7270-4561-9e62-15b3950eb8c9"). InnerVolumeSpecName "kube-api-access-74q2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:43.462195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.462173 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:56:43.529891 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.529861 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\") pod \"081e4e07-8050-499a-8f1f-0771a825bf4d\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " Nov 03 22:56:43.530084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.529928 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\" (UniqueName: \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\") pod \"081e4e07-8050-499a-8f1f-0771a825bf4d\" (UID: \"081e4e07-8050-499a-8f1f-0771a825bf4d\") " Nov 03 22:56:43.530332 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.530301 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume" pod "081e4e07-8050-499a-8f1f-0771a825bf4d" (UID: "081e4e07-8050-499a-8f1f-0771a825bf4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:43.530775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.530727 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e4e07-8050-499a-8f1f-0771a825bf4d-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:43.530775 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.530751 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q2p\" (UniqueName: \"kubernetes.io/projected/ca99ec5e-7270-4561-9e62-15b3950eb8c9-kube-api-access-74q2p\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:43.532557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.532508 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm" pod "081e4e07-8050-499a-8f1f-0771a825bf4d" (UID: "081e4e07-8050-499a-8f1f-0771a825bf4d"). InnerVolumeSpecName "kube-api-access-b6nfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:43.631759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:43.631718 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nfm\" (UniqueName: \"kubernetes.io/projected/081e4e07-8050-499a-8f1f-0771a825bf4d-kube-api-access-b6nfm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:44.166304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.166266 5051 generic.go:356] "Generic (PLEG): container finished" podID="081e4e07-8050-499a-8f1f-0771a825bf4d" containerID="c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f" exitCode=0 Nov 03 22:56:44.166759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.166309 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" event={"ID":"081e4e07-8050-499a-8f1f-0771a825bf4d","Type":"ContainerDied","Data":"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f"} Nov 03 22:56:44.166759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.166378 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" event={"ID":"081e4e07-8050-499a-8f1f-0771a825bf4d","Type":"ContainerDied","Data":"4ebb92c476c45b33ceaf3cd26db7b9c11ad33af701a0b01883e6b2ec586208c8"} Nov 03 22:56:44.166759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.166408 5051 scope.go:122] "RemoveContainer" containerID="c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f" Nov 03 22:56:44.166759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.166319 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw" Nov 03 22:56:44.168281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.168257 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" event={"ID":"ca99ec5e-7270-4561-9e62-15b3950eb8c9","Type":"ContainerDied","Data":"4c24d0939df72e217ca330323d65806998cc7453df4bcd5494fc66aa44feefee"} Nov 03 22:56:44.168281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.168274 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl" Nov 03 22:56:44.180050 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.180033 5051 scope.go:122] "RemoveContainer" containerID="c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f" Nov 03 22:56:44.180313 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:44.180289 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f\": container with ID starting with c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f not found: ID does not exist" containerID="c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f" Nov 03 22:56:44.180369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.180321 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f"} err="failed to get container status \"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f\": rpc error: code = NotFound desc = could not find container \"c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f\": container with ID starting with c94f1b142cad38528cfd2654f50ce1f5ecb6254f7ec14eeb4ae9d3fc36dcbc4f not found: ID does not exist" Nov 03 22:56:44.180369 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.180342 5051 scope.go:122] "RemoveContainer" containerID="fc6ff0a921593e13ddeb5d83e1239fdae4b7bee7a7ea86e3f4b17797a90ebb51" Nov 03 22:56:44.332794 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.332713 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:56:44.381935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.381895 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a91d2d-c8f444867-j8llw"] Nov 03 22:56:44.411973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.411945 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081e4e07-8050-499a-8f1f-0771a825bf4d" path="/var/lib/kubelet/pods/081e4e07-8050-499a-8f1f-0771a825bf4d/volumes" Nov 03 22:56:44.444670 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.444638 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:56:44.450697 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:44.450674 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a91d2d-7b5d5fbfbb-k5gzl"] Nov 03 22:56:45.720849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:45.720811 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" podUID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" containerName="payload-pre-processing" containerID="cri-o://3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9" gracePeriod=25 Nov 03 22:56:46.018364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.018339 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:56:46.103686 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.103642 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" podUID="cd44189a-405c-4974-b08b-e71ebb0ceb20" containerName="payload-processing" containerID="cri-o://997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7" gracePeriod=25 Nov 03 22:56:46.168349 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.168319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\") pod \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " Nov 03 22:56:46.168558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.168451 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\" (UniqueName: \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\") pod \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\" (UID: \"78d5786e-46b8-4ae6-957c-ccef18eb39b3\") " Nov 03 22:56:46.169277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.168860 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume" pod "78d5786e-46b8-4ae6-957c-ccef18eb39b3" (UID: "78d5786e-46b8-4ae6-957c-ccef18eb39b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:46.169277 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.169236 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5786e-46b8-4ae6-957c-ccef18eb39b3-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.172040 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.171254 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8" pod "78d5786e-46b8-4ae6-957c-ccef18eb39b3" (UID: "78d5786e-46b8-4ae6-957c-ccef18eb39b3"). InnerVolumeSpecName "kube-api-access-7nlc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:46.193517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.193482 5051 generic.go:356] "Generic (PLEG): container finished" podID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" containerID="3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9" exitCode=0 Nov 03 22:56:46.193645 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.193616 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" Nov 03 22:56:46.193719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.193632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" event={"ID":"78d5786e-46b8-4ae6-957c-ccef18eb39b3","Type":"ContainerDied","Data":"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9"} Nov 03 22:56:46.193719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.193702 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c" event={"ID":"78d5786e-46b8-4ae6-957c-ccef18eb39b3","Type":"ContainerDied","Data":"41a8e04a5e2a53049a0d65b0d199821fcbd7f40ec042814e169ef8c157e8415d"} Nov 03 22:56:46.193831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.193733 5051 scope.go:122] "RemoveContainer" containerID="3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9" Nov 03 22:56:46.218162 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.218141 5051 scope.go:122] "RemoveContainer" containerID="3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9" Nov 03 22:56:46.218553 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:46.218530 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9\": container with ID starting with 3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9 not found: ID does not exist" containerID="3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9" Nov 03 22:56:46.218664 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.218559 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9"} err="failed to get container status \"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9\": rpc error: code = NotFound desc = could not find container \"3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9\": container with ID starting with 3b27b473293e19a2fd60f694e25da42610b89f4ec882de6901ba772cfe0e96b9 not found: ID does not exist" Nov 03 22:56:46.271193 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.271119 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlc8\" (UniqueName: \"kubernetes.io/projected/78d5786e-46b8-4ae6-957c-ccef18eb39b3-kube-api-access-7nlc8\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.369284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.367103 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:56:46.373316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.373289 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-de5715-89cc48587-2nf7c"] Nov 03 22:56:46.414401 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.414372 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" path="/var/lib/kubelet/pods/78d5786e-46b8-4ae6-957c-ccef18eb39b3/volumes" Nov 03 22:56:46.414884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.414867 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" path="/var/lib/kubelet/pods/ca99ec5e-7270-4561-9e62-15b3950eb8c9/volumes" Nov 03 22:56:46.425575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.425555 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:56:46.595591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.580658 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\" (UniqueName: \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\") pod \"cd44189a-405c-4974-b08b-e71ebb0ceb20\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " Nov 03 22:56:46.595591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.580803 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\") pod \"cd44189a-405c-4974-b08b-e71ebb0ceb20\" (UID: \"cd44189a-405c-4974-b08b-e71ebb0ceb20\") " Nov 03 22:56:46.595591 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.582116 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume" pod "cd44189a-405c-4974-b08b-e71ebb0ceb20" (UID: "cd44189a-405c-4974-b08b-e71ebb0ceb20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:46.652251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.652211 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9" pod "cd44189a-405c-4974-b08b-e71ebb0ceb20" (UID: "cd44189a-405c-4974-b08b-e71ebb0ceb20"). InnerVolumeSpecName "kube-api-access-htkl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:46.689172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.686705 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkl9\" (UniqueName: \"kubernetes.io/projected/cd44189a-405c-4974-b08b-e71ebb0ceb20-kube-api-access-htkl9\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.689445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:46.689407 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd44189a-405c-4974-b08b-e71ebb0ceb20-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:47.220171 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.220134 5051 generic.go:356] "Generic (PLEG): container finished" podID="cd44189a-405c-4974-b08b-e71ebb0ceb20" containerID="997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7" exitCode=0 Nov 03 22:56:47.220661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.220207 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" Nov 03 22:56:47.220661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.220217 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" event={"ID":"cd44189a-405c-4974-b08b-e71ebb0ceb20","Type":"ContainerDied","Data":"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7"} Nov 03 22:56:47.220661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.220324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8" event={"ID":"cd44189a-405c-4974-b08b-e71ebb0ceb20","Type":"ContainerDied","Data":"18ae576d98fdf0d4f113fb801cd36c378ce189d1f4c6793c3548cb1c2bb10573"} Nov 03 22:56:47.220661 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.220352 5051 scope.go:122] "RemoveContainer" containerID="997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7" Nov 03 22:56:47.235053 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.235031 5051 scope.go:122] "RemoveContainer" containerID="997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7" Nov 03 22:56:47.235377 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:47.235354 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7\": container with ID starting with 997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7 not found: ID does not exist" containerID="997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7" Nov 03 22:56:47.235471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.235388 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7"} err="failed to get container status \"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7\": rpc error: code = NotFound desc = could not find container \"997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7\": container with ID starting with 997b17042c0484defbe6bc97fbe6ce7a244b976e3fa5decce2544616846bc2c7 not found: ID does not exist" Nov 03 22:56:47.402725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.402695 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:56:47.407179 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:47.407149 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-de5715-7c6c757c54-cr9p8"] Nov 03 22:56:48.396154 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:48.396120 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn"] Nov 03 22:56:48.409387 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:48.409346 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd44189a-405c-4974-b08b-e71ebb0ceb20" path="/var/lib/kubelet/pods/cd44189a-405c-4974-b08b-e71ebb0ceb20/volumes" Nov 03 22:56:48.438069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:48.438042 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-efce83-zfrcn"] Nov 03 22:56:49.527298 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:49.527261 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5"] Nov 03 22:56:49.576137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:49.576110 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-zntp5"] Nov 03 22:56:50.417451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:50.417413 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee" path="/var/lib/kubelet/pods/d5e3f2e5-abc3-41ec-a8bf-2abaa7d7aeee/volumes" Nov 03 22:56:50.418862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:50.418829 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66451bf-8657-4eb0-b592-b554812ae308" path="/var/lib/kubelet/pods/f66451bf-8657-4eb0-b592-b554812ae308/volumes" Nov 03 22:56:51.624853 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:56:51.624813 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:56:58.875062 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875011 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 22:56:58.875900 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875874 5051 cpu_manager.go:414] "containerMap: removing container" podUID="081e4e07-8050-499a-8f1f-0771a825bf4d" containerName="payload-pre-processing" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875901 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081e4e07-8050-499a-8f1f-0771a825bf4d" containerName="payload-pre-processing" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875929 5051 cpu_manager.go:414] "containerMap: removing container" podUID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" containerName="payload-pre-processing" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875936 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" containerName="payload-pre-processing" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875965 5051 cpu_manager.go:414] "containerMap: removing container" podUID="857d556c-76e0-424a-b0dc-4fa1e7f1066c" containerName="revoke-keys" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.875973 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857d556c-76e0-424a-b0dc-4fa1e7f1066c" containerName="revoke-keys" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876014 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5b427274-491f-4a08-9e61-406e24df1797" containerName="revoke-keys" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876022 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b427274-491f-4a08-9e61-406e24df1797" containerName="revoke-keys" Nov 03 22:56:58.876036 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876039 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cd44189a-405c-4974-b08b-e71ebb0ceb20" containerName="payload-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876046 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd44189a-405c-4974-b08b-e71ebb0ceb20" containerName="payload-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876059 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" containerName="payload-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876066 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" containerName="payload-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876295 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="081e4e07-8050-499a-8f1f-0771a825bf4d" containerName="payload-pre-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876312 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d5786e-46b8-4ae6-957c-ccef18eb39b3" containerName="payload-pre-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876326 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="857d556c-76e0-424a-b0dc-4fa1e7f1066c" containerName="revoke-keys" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876338 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b427274-491f-4a08-9e61-406e24df1797" containerName="revoke-keys" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876350 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd44189a-405c-4974-b08b-e71ebb0ceb20" containerName="payload-processing" Nov 03 22:56:58.876441 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.876359 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca99ec5e-7270-4561-9e62-15b3950eb8c9" containerName="payload-processing" Nov 03 22:56:58.880284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.880261 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:58.997786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:58.941242 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-efd39a-openshift-default-dockercfg-rmxkh\"" Nov 03 22:56:59.071804 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.053358 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslwp\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.072128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.072099 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.108015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.106083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.128365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.110173 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.128365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.126254 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 22:56:59.128821 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.128794 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.129104 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.129086 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.129255 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.129241 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.129488 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.129474 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.129634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.129618 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233478 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233589 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233648 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233687 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233817 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.236769 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.237054 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.237687 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.237897 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.233857 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.238204 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.238300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslwp\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.238328 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.238354 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.239096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.240358 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.239763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.299516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.299484 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.299721 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.299698 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslwp\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\") pod \"e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.497372 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.497282 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:56:59.832759 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.832614 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:56:59.856359 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.838062 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:56:59.890221 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.890191 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a9ab23c4\"" Nov 03 22:56:59.890725 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.890466 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a9ab23c4-dockercfg-zxrw7\"" Nov 03 22:56:59.895399 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.895373 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:56:59.996013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.981966 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:56:59.996013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:56:59.982116 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbwc\" (UniqueName: \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.037014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.034979 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:57:00.062069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.061676 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.101519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.101137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.101519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.101360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbwc\" (UniqueName: \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.122014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.103295 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.129437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.129355 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:57:00.143379 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.143351 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 22:57:00.156464 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:00.156437 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98725bfa_0f3c_4e0a_8ddf_3f8cb56974ae.slice/crio-abb1573fab9ef4e18e61b1ef459cdc275c996a168835ce0237c3cdd7ad4768b1 WatchSource:0}: Error finding container abb1573fab9ef4e18e61b1ef459cdc275c996a168835ce0237c3cdd7ad4768b1: Status 404 returned error can't find the container with id abb1573fab9ef4e18e61b1ef459cdc275c996a168835ce0237c3cdd7ad4768b1 Nov 03 22:57:00.172952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.159696 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:57:00.172952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.159781 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:57:00.172952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.159820 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:57:00.177936 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.176340 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbwc\" (UniqueName: \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\") pod \"payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.202940 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.202902 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.203092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.202967 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hbb\" (UniqueName: \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.304768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.304329 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.304768 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.304389 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hbb\" (UniqueName: \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.305292 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.305269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.365577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.365545 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hbb\" (UniqueName: \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\") pod \"payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.438347 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.438315 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:00.454253 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.454220 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:00.520574 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.520532 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerStarted","Data":"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27"} Nov 03 22:57:00.520900 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.520602 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerStarted","Data":"abb1573fab9ef4e18e61b1ef459cdc275c996a168835ce0237c3cdd7ad4768b1"} Nov 03 22:57:00.654188 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.654068 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podStartSLOduration=2.654047924 podStartE2EDuration="2.654047924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:00.646982239 +0000 UTC m=+2094.384326469" watchObservedRunningTime="2026-11-03 22:57:00.654047924 +0000 UTC m=+2094.391392149" Nov 03 22:57:00.866106 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:00.866077 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd891fb36_0e67_4dbf_a1b4_b6228912d368.slice/crio-a7412fc226991fc10d07d992dcc0cc040cb6debdcd8f3d9ef6b55318a0bc8717 WatchSource:0}: Error finding container a7412fc226991fc10d07d992dcc0cc040cb6debdcd8f3d9ef6b55318a0bc8717: Status 404 returned error can't find the container with id a7412fc226991fc10d07d992dcc0cc040cb6debdcd8f3d9ef6b55318a0bc8717 Nov 03 22:57:00.867313 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.867281 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:57:00.911455 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:00.911428 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:57:00.912184 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:00.911831 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269bf59_fe87_4e27_a265_5e377cd60aa2.slice/crio-94f3742748134ed30f2a14bfec7d0566afa79c9423d8209a14a0101f8e679aba WatchSource:0}: Error finding container 94f3742748134ed30f2a14bfec7d0566afa79c9423d8209a14a0101f8e679aba: Status 404 returned error can't find the container with id 94f3742748134ed30f2a14bfec7d0566afa79c9423d8209a14a0101f8e679aba Nov 03 22:57:01.498133 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.498098 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:57:01.504166 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.504133 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:57:01.531696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.531659 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" event={"ID":"d891fb36-0e67-4dbf-a1b4-b6228912d368","Type":"ContainerStarted","Data":"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b"} Nov 03 22:57:01.531883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.531719 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" event={"ID":"d891fb36-0e67-4dbf-a1b4-b6228912d368","Type":"ContainerStarted","Data":"a7412fc226991fc10d07d992dcc0cc040cb6debdcd8f3d9ef6b55318a0bc8717"} Nov 03 22:57:01.531883 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.531747 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:01.533637 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.533609 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" event={"ID":"8269bf59-fe87-4e27-a265-5e377cd60aa2","Type":"ContainerStarted","Data":"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017"} Nov 03 22:57:01.533798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.533656 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" event={"ID":"8269bf59-fe87-4e27-a265-5e377cd60aa2","Type":"ContainerStarted","Data":"94f3742748134ed30f2a14bfec7d0566afa79c9423d8209a14a0101f8e679aba"} Nov 03 22:57:01.533798 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.533699 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:01.534212 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.534198 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:57:01.535106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.535086 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:57:01.760689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.760570 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" podStartSLOduration=2.760552302 podStartE2EDuration="2.760552302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:01.760359491 +0000 UTC m=+2095.497703719" watchObservedRunningTime="2026-11-03 22:57:01.760552302 +0000 UTC m=+2095.497896528" Nov 03 22:57:01.880590 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:01.880527 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" podStartSLOduration=2.880508312 podStartE2EDuration="2.880508312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:01.874718357 +0000 UTC m=+2095.612062602" watchObservedRunningTime="2026-11-03 22:57:01.880508312 +0000 UTC m=+2095.617852538" Nov 03 22:57:07.145015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:07.141242 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:57:07.166785 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:07.166757 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 22:57:09.444491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:09.444439 5051 scope.go:122] "RemoveContainer" containerID="246965c6eb4cb3649970b01d22798e4bdbf19f99a769c019c4c8879d681c9e02" Nov 03 22:57:11.563343 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:11.563311 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb"] Nov 03 22:57:11.612383 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:11.612345 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-26444f-r6kwb"] Nov 03 22:57:12.409968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:12.409934 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca03d21-3462-4120-a1cc-363614cf0545" path="/var/lib/kubelet/pods/cca03d21-3462-4120-a1cc-363614cf0545/volumes" Nov 03 22:57:12.545904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:12.545875 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:57:12.546871 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:12.546845 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:57:21.657729 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:57:21.657637 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:57:22.475244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:22.475206 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk"] Nov 03 22:57:22.523772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:22.523738 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ea1be0-lwzhk"] Nov 03 22:57:24.413542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:24.413511 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79629b14-d38c-41e3-8892-7301e4aa5a4b" path="/var/lib/kubelet/pods/79629b14-d38c-41e3-8892-7301e4aa5a4b/volumes" Nov 03 22:57:29.778485 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.778451 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:57:29.778979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.778635 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" containerID="cri-o://865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15" gracePeriod=30 Nov 03 22:57:29.782793 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.782769 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 22:57:29.787202 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.787183 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:29.811621 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.811594 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 22:57:29.857962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.857925 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f8cd0c\"" Nov 03 22:57:29.922021 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.881398 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f8cd0c-dockercfg-kc29r\"" Nov 03 22:57:29.924029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.922760 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj9k\" (UniqueName: \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:29.924029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:29.922850 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.024484 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.024447 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj9k\" (UniqueName: \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.024808 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.024783 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.025790 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.025768 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.085807 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.085734 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj9k\" (UniqueName: \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\") pod \"payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.102378 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.102350 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:30.484722 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.484686 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 22:57:30.486090 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:30.486066 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3146172_606a_4acb_a24c_8c24a85a72df.slice/crio-3a5426c6292af35f10fb0e3d1fb55538997b7e430e48fd604117771e84788454 WatchSource:0}: Error finding container 3a5426c6292af35f10fb0e3d1fb55538997b7e430e48fd604117771e84788454: Status 404 returned error can't find the container with id 3a5426c6292af35f10fb0e3d1fb55538997b7e430e48fd604117771e84788454 Nov 03 22:57:30.684461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.684431 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647"] Nov 03 22:57:30.688962 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.688936 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.742090 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.741982 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:30.742333 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.742227 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:57:30.747265 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.747234 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647"] Nov 03 22:57:30.836770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.836731 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ndd\" (UniqueName: \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.837348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.836918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.837348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.836939 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.938509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.938475 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.938509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.938515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.938734 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.938670 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ndd\" (UniqueName: \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.939480 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.939449 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:30.941176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:30.941152 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:31.000365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.000265 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ndd\" (UniqueName: \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:31.034647 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.025229 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:31.106920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.106092 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" event={"ID":"c3146172-606a-4acb-a24c-8c24a85a72df","Type":"ContainerStarted","Data":"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1"} Nov 03 22:57:31.106920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.106186 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:31.106920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.106213 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" event={"ID":"c3146172-606a-4acb-a24c-8c24a85a72df","Type":"ContainerStarted","Data":"3a5426c6292af35f10fb0e3d1fb55538997b7e430e48fd604117771e84788454"} Nov 03 22:57:31.223718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.223670 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" podStartSLOduration=2.223655417 podStartE2EDuration="2.223655417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:31.221565099 +0000 UTC m=+2124.958909324" watchObservedRunningTime="2026-11-03 22:57:31.223655417 +0000 UTC m=+2124.960999639" Nov 03 22:57:31.438967 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:31.438936 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647"] Nov 03 22:57:31.440076 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:31.440027 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1812bbeb_24b2_44c5_a34e_2289476123fa.slice/crio-ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec WatchSource:0}: Error finding container ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec: Status 404 returned error can't find the container with id ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec Nov 03 22:57:32.115983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:32.115948 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" event={"ID":"1812bbeb-24b2-44c5-a34e-2289476123fa","Type":"ContainerStarted","Data":"d0143c2bb1a42b3a6ade86ac303ca38c90032832c190544732c9c6934d3b408c"} Nov 03 22:57:32.116465 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:32.116023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" event={"ID":"1812bbeb-24b2-44c5-a34e-2289476123fa","Type":"ContainerStarted","Data":"ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec"} Nov 03 22:57:32.231352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:32.231284 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" podStartSLOduration=2.231270547 podStartE2EDuration="2.231270547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:32.230788098 +0000 UTC m=+2125.968132323" watchObservedRunningTime="2026-11-03 22:57:32.231270547 +0000 UTC m=+2125.968614783" Nov 03 22:57:33.125812 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:33.125774 5051 generic.go:356] "Generic (PLEG): container finished" podID="1812bbeb-24b2-44c5-a34e-2289476123fa" containerID="d0143c2bb1a42b3a6ade86ac303ca38c90032832c190544732c9c6934d3b408c" exitCode=0 Nov 03 22:57:33.126328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:33.125881 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" event={"ID":"1812bbeb-24b2-44c5-a34e-2289476123fa","Type":"ContainerDied","Data":"d0143c2bb1a42b3a6ade86ac303ca38c90032832c190544732c9c6934d3b408c"} Nov 03 22:57:34.322326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.322299 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:34.495252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.495161 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\" (UniqueName: \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\") pod \"1812bbeb-24b2-44c5-a34e-2289476123fa\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " Nov 03 22:57:34.495252 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.495217 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\") pod \"1812bbeb-24b2-44c5-a34e-2289476123fa\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " Nov 03 22:57:34.495482 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.495319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\") pod \"1812bbeb-24b2-44c5-a34e-2289476123fa\" (UID: \"1812bbeb-24b2-44c5-a34e-2289476123fa\") " Nov 03 22:57:34.497198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.495973 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca" pod "1812bbeb-24b2-44c5-a34e-2289476123fa" (UID: "1812bbeb-24b2-44c5-a34e-2289476123fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:34.498316 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.498289 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca" pod "1812bbeb-24b2-44c5-a34e-2289476123fa" (UID: "1812bbeb-24b2-44c5-a34e-2289476123fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:34.498460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.498333 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd" pod "1812bbeb-24b2-44c5-a34e-2289476123fa" (UID: "1812bbeb-24b2-44c5-a34e-2289476123fa"). InnerVolumeSpecName "kube-api-access-n9ndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:34.598240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.598194 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ndd\" (UniqueName: \"kubernetes.io/projected/1812bbeb-24b2-44c5-a34e-2289476123fa-kube-api-access-n9ndd\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:34.598240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.598233 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:34.598240 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:34.598244 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1812bbeb-24b2-44c5-a34e-2289476123fa-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.049861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.049796 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:57:35.144194 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.144163 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" Nov 03 22:57:35.144394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.144202 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647" event={"ID":"1812bbeb-24b2-44c5-a34e-2289476123fa","Type":"ContainerDied","Data":"ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec"} Nov 03 22:57:35.144394 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.144248 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab778e85953440f6fd88f53bbfa2d8ae12136db2b347fcd1f408ba9fe32595ec" Nov 03 22:57:35.146304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.146278 5051 generic.go:356] "Generic (PLEG): container finished" podID="83820620-a67c-4073-a7e7-3d05224575a8" containerID="865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15" exitCode=0 Nov 03 22:57:35.146468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.146334 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" Nov 03 22:57:35.146468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.146366 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerDied","Data":"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15"} Nov 03 22:57:35.146468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.146410 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg" event={"ID":"83820620-a67c-4073-a7e7-3d05224575a8","Type":"ContainerDied","Data":"dc4a0a16a45a77b8a35ef48566f2aad0d4e8a863040c25d61167937c9ef8a2a8"} Nov 03 22:57:35.146468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.146434 5051 scope.go:122] "RemoveContainer" containerID="865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15" Nov 03 22:57:35.158208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.158192 5051 scope.go:122] "RemoveContainer" containerID="b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557" Nov 03 22:57:35.169491 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.169467 5051 scope.go:122] "RemoveContainer" containerID="865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15" Nov 03 22:57:35.169757 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:57:35.169739 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15\": container with ID starting with 865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15 not found: ID does not exist" containerID="865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15" Nov 03 22:57:35.169818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.169765 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15"} err="failed to get container status \"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15\": rpc error: code = NotFound desc = could not find container \"865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15\": container with ID starting with 865ca9898d6f295d238c95cef76a6178cf4fa1951f4ce078e568994ba3e6ee15 not found: ID does not exist" Nov 03 22:57:35.169818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.169780 5051 scope.go:122] "RemoveContainer" containerID="b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557" Nov 03 22:57:35.170042 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:57:35.170024 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557\": container with ID starting with b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557 not found: ID does not exist" containerID="b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557" Nov 03 22:57:35.170088 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.170046 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557"} err="failed to get container status \"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557\": rpc error: code = NotFound desc = could not find container \"b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557\": container with ID starting with b2241a47b9adbffb3d0c62ffcdc2b09351e6bbaffb020841dccc619947e50557 not found: ID does not exist" Nov 03 22:57:35.205813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.205780 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.205835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.205869 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.205922 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206065 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206149 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206174 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:35.206232 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206178 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206276 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206337 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:35.206460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206371 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:35.206460 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206430 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\") pod \"83820620-a67c-4073-a7e7-3d05224575a8\" (UID: \"83820620-a67c-4073-a7e7-3d05224575a8\") " Nov 03 22:57:35.206665 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206485 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:35.206792 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.206770 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:35.207461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.207422 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.207461 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.207454 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.207623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.207469 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.207623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.207484 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.207623 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.207498 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83820620-a67c-4073-a7e7-3d05224575a8-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.208340 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.208319 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:35.208493 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.208475 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:57:35.208578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.208528 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:35.208897 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.208872 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq" pod "83820620-a67c-4073-a7e7-3d05224575a8" (UID: "83820620-a67c-4073-a7e7-3d05224575a8"). InnerVolumeSpecName "kube-api-access-rj5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:35.308843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.308764 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83820620-a67c-4073-a7e7-3d05224575a8-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.308843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.308796 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83820620-a67c-4073-a7e7-3d05224575a8-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.308843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.308809 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.308843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.308821 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj5sq\" (UniqueName: \"kubernetes.io/projected/83820620-a67c-4073-a7e7-3d05224575a8-kube-api-access-rj5sq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:35.572573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.572467 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:57:35.576776 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:35.576748 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ea1be0-openshift-default-6bfb8dcf95-wl6kg"] Nov 03 22:57:36.413013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:36.412968 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83820620-a67c-4073-a7e7-3d05224575a8" path="/var/lib/kubelet/pods/83820620-a67c-4073-a7e7-3d05224575a8/volumes" Nov 03 22:57:39.027698 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:39.027664 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx"] Nov 03 22:57:39.079294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:39.079265 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-de5715-q2lrx"] Nov 03 22:57:40.410884 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:40.410852 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8" path="/var/lib/kubelet/pods/a6870f66-b1c2-48f0-b9cb-f8e1e88f11e8/volumes" Nov 03 22:57:42.120089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:42.120054 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 22:57:42.300314 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:42.300283 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:57:46.733732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.733697 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb"] Nov 03 22:57:46.734614 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734591 5051 cpu_manager.go:414] "containerMap: removing container" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:57:46.734738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734615 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:57:46.734738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734695 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1812bbeb-24b2-44c5-a34e-2289476123fa" containerName="revoke-keys" Nov 03 22:57:46.734738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734704 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1812bbeb-24b2-44c5-a34e-2289476123fa" containerName="revoke-keys" Nov 03 22:57:46.734904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734879 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1812bbeb-24b2-44c5-a34e-2289476123fa" containerName="revoke-keys" Nov 03 22:57:46.734904 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734893 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:57:46.735022 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.734906 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:57:46.740203 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.740182 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.797010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.796484 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb"] Nov 03 22:57:46.811260 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.811229 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.811466 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.811449 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.811560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.811544 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.838612 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.838585 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:46.838859 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.838840 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:57:46.913294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.913257 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.913463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.913331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.913463 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.913446 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.914073 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.914051 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.916117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.916096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:46.976903 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:46.976871 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:47.055172 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.055100 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:47.301453 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.301383 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" podUID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" containerName="payload-pre-processing" containerID="cri-o://da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285" gracePeriod=25 Nov 03 22:57:47.454683 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.454277 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb"] Nov 03 22:57:47.463038 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:57:47.457141 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e727ae_5c2e_40f6_bfff_e4d5c7967140.slice/crio-eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb WatchSource:0}: Error finding container eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb: Status 404 returned error can't find the container with id eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb Nov 03 22:57:47.587047 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.586979 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:57:47.729127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.729085 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\" (UniqueName: \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\") pod \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " Nov 03 22:57:47.729468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.729439 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\") pod \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\" (UID: \"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f\") " Nov 03 22:57:47.729887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.729862 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume" pod "02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" (UID: "02627d5e-2e7b-4f2e-b2bf-c49b18248a8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:47.731814 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.731784 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:47.732138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.732115 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88" pod "02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" (UID: "02627d5e-2e7b-4f2e-b2bf-c49b18248a8f"). InnerVolumeSpecName "kube-api-access-bmd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:47.832952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:47.832869 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmd88\" (UniqueName: \"kubernetes.io/projected/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f-kube-api-access-bmd88\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:48.259404 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.259369 5051 generic.go:356] "Generic (PLEG): container finished" podID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" containerID="da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285" exitCode=0 Nov 03 22:57:48.259708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.259449 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" event={"ID":"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f","Type":"ContainerDied","Data":"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285"} Nov 03 22:57:48.259708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.259499 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" event={"ID":"02627d5e-2e7b-4f2e-b2bf-c49b18248a8f","Type":"ContainerDied","Data":"bd9aa2ce1b6933656887ae53cddd526cd1079f88c4820df63216a89e03af33fd"} Nov 03 22:57:48.259708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.259530 5051 scope.go:122] "RemoveContainer" containerID="da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285" Nov 03 22:57:48.259708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.259678 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j" Nov 03 22:57:48.262961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.262930 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" event={"ID":"09e727ae-5c2e-40f6-bfff-e4d5c7967140","Type":"ContainerStarted","Data":"c22a69f4d0681b3aebfb36a62b2fa6fc4c917585ee4a9df9b3c20ee59572ab4e"} Nov 03 22:57:48.263150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.263018 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" event={"ID":"09e727ae-5c2e-40f6-bfff-e4d5c7967140","Type":"ContainerStarted","Data":"eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb"} Nov 03 22:57:48.272888 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.272869 5051 scope.go:122] "RemoveContainer" containerID="da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285" Nov 03 22:57:48.273209 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:57:48.273182 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285\": container with ID starting with da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285 not found: ID does not exist" containerID="da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285" Nov 03 22:57:48.273294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.273218 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285"} err="failed to get container status \"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285\": rpc error: code = NotFound desc = could not find container \"da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285\": container with ID starting with da1a0f430b2e6734d30ebe367b3c4813c66f5ee96f567905c9fd0dcdc7166285 not found: ID does not exist" Nov 03 22:57:48.429181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.426327 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:57:48.434639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.433123 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0406a8-648f9b947-9vn4j"] Nov 03 22:57:48.540447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:48.540352 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" podStartSLOduration=2.54033772 podStartE2EDuration="2.54033772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:48.539285424 +0000 UTC m=+2142.276629654" watchObservedRunningTime="2026-11-03 22:57:48.54033772 +0000 UTC m=+2142.277681945" Nov 03 22:57:49.275770 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:49.275730 5051 generic.go:356] "Generic (PLEG): container finished" podID="09e727ae-5c2e-40f6-bfff-e4d5c7967140" containerID="c22a69f4d0681b3aebfb36a62b2fa6fc4c917585ee4a9df9b3c20ee59572ab4e" exitCode=0 Nov 03 22:57:49.276189 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:49.275813 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" event={"ID":"09e727ae-5c2e-40f6-bfff-e4d5c7967140","Type":"ContainerDied","Data":"c22a69f4d0681b3aebfb36a62b2fa6fc4c917585ee4a9df9b3c20ee59572ab4e"} Nov 03 22:57:50.422456 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.422425 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" path="/var/lib/kubelet/pods/02627d5e-2e7b-4f2e-b2bf-c49b18248a8f/volumes" Nov 03 22:57:50.467941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.467918 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:50.573737 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.573588 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\") pod \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " Nov 03 22:57:50.573911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.573868 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca" pod "09e727ae-5c2e-40f6-bfff-e4d5c7967140" (UID: "09e727ae-5c2e-40f6-bfff-e4d5c7967140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:50.573911 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.573871 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\") pod \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " Nov 03 22:57:50.574070 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.573964 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\") pod \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\" (UID: \"09e727ae-5c2e-40f6-bfff-e4d5c7967140\") " Nov 03 22:57:50.574745 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.574723 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:50.576760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.576729 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca" pod "09e727ae-5c2e-40f6-bfff-e4d5c7967140" (UID: "09e727ae-5c2e-40f6-bfff-e4d5c7967140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:50.576760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.576750 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb" pod "09e727ae-5c2e-40f6-bfff-e4d5c7967140" (UID: "09e727ae-5c2e-40f6-bfff-e4d5c7967140"). InnerVolumeSpecName "kube-api-access-dhcbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:50.676611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.676577 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09e727ae-5c2e-40f6-bfff-e4d5c7967140-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:50.676611 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:50.676612 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcbb\" (UniqueName: \"kubernetes.io/projected/09e727ae-5c2e-40f6-bfff-e4d5c7967140-kube-api-access-dhcbb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:51.296548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:51.296515 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" Nov 03 22:57:51.296548 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:51.296533 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb" event={"ID":"09e727ae-5c2e-40f6-bfff-e4d5c7967140","Type":"ContainerDied","Data":"eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb"} Nov 03 22:57:51.296802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:57:51.296612 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8d1a593bed6b95aa2ce0bf8f62e7da078553adf08424986a58249c142f84fb" Nov 03 22:57:51.625886 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:57:51.625850 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:58:00.514084 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:00.514049 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5"] Nov 03 22:58:00.565961 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:00.565928 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a91d2d-5qnw5"] Nov 03 22:58:02.409954 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:02.409926 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857d556c-76e0-424a-b0dc-4fa1e7f1066c" path="/var/lib/kubelet/pods/857d556c-76e0-424a-b0dc-4fa1e7f1066c/volumes" Nov 03 22:58:10.703000 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.702954 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf"] Nov 03 22:58:10.704039 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704017 5051 cpu_manager.go:414] "containerMap: removing container" podUID="09e727ae-5c2e-40f6-bfff-e4d5c7967140" containerName="revoke-keys" Nov 03 22:58:10.704100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704042 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e727ae-5c2e-40f6-bfff-e4d5c7967140" containerName="revoke-keys" Nov 03 22:58:10.704100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704088 5051 cpu_manager.go:414] "containerMap: removing container" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:58:10.704100 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704097 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83820620-a67c-4073-a7e7-3d05224575a8" containerName="istio-proxy" Nov 03 22:58:10.704195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704132 5051 cpu_manager.go:414] "containerMap: removing container" podUID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" containerName="payload-pre-processing" Nov 03 22:58:10.704195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704140 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" containerName="payload-pre-processing" Nov 03 22:58:10.704357 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704345 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="02627d5e-2e7b-4f2e-b2bf-c49b18248a8f" containerName="payload-pre-processing" Nov 03 22:58:10.704396 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.704370 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e727ae-5c2e-40f6-bfff-e4d5c7967140" containerName="revoke-keys" Nov 03 22:58:10.709965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.709948 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.762584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.762544 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:10.762968 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.762947 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:58:10.764706 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.764683 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf"] Nov 03 22:58:10.827502 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.827465 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.827671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.827650 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppgc\" (UniqueName: \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.827828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.827811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.929585 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.929495 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.929772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.929622 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppgc\" (UniqueName: \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.929772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.929763 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.930326 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.930306 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.932328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.932310 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:10.990509 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:10.990477 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppgc\" (UniqueName: \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\") pod \"maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:11.025067 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:11.025039 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:11.413813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:11.413782 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf"] Nov 03 22:58:11.415344 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:58:11.415314 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2703b0a9_4a81_4924_9f1d_6ea9a7d07c7e.slice/crio-0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139 WatchSource:0}: Error finding container 0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139: Status 404 returned error can't find the container with id 0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139 Nov 03 22:58:11.478952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:11.478921 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" event={"ID":"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e","Type":"ContainerStarted","Data":"0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139"} Nov 03 22:58:12.489360 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:12.489271 5051 generic.go:356] "Generic (PLEG): container finished" podID="2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" containerID="f87a078673a144783ac7f79805a73558087c485a23410b27160c43b7a4202a39" exitCode=0 Nov 03 22:58:12.489861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:12.489358 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" event={"ID":"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e","Type":"ContainerDied","Data":"f87a078673a144783ac7f79805a73558087c485a23410b27160c43b7a4202a39"} Nov 03 22:58:13.713843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.713807 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:13.781270 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.781193 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\") pod \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " Nov 03 22:58:13.781442 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.781280 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\") pod \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " Nov 03 22:58:13.781531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.781494 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\" (UniqueName: \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\") pod \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\" (UID: \"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e\") " Nov 03 22:58:13.781640 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.781615 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca" pod "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" (UID: "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:13.781952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.781934 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.783558 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.783536 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca" pod "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" (UID: "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:13.783738 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.783724 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc" pod "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" (UID: "2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e"). InnerVolumeSpecName "kube-api-access-zppgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:13.884186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.884146 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zppgc\" (UniqueName: \"kubernetes.io/projected/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-kube-api-access-zppgc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.884352 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:13.884191 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:14.507354 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:14.507325 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" Nov 03 22:58:14.507519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:14.507354 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf" event={"ID":"2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e","Type":"ContainerDied","Data":"0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139"} Nov 03 22:58:14.507519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:14.507412 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d685d4dc19a0a6ab23841eeac4b35d7dad729939107cfe0e993fdf724fe8139" Nov 03 22:58:21.625250 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:58:21.625207 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:58:41.498346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:41.498302 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:58:41.499957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:41.499913 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" containerID="cri-o://7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d" gracePeriod=30 Nov 03 22:58:41.619643 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:41.619606 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:58:41.619978 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:41.619929 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" containerID="cri-o://15abb30c357316f21974c3b2fa97c98ebc9604d9594e3acadf50a7a03d9c3cfc" gracePeriod=30 Nov 03 22:58:46.549004 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.548949 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:46.549420 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.549059 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:46.805732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.805380 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:58:46.816017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.815959 5051 generic.go:356] "Generic (PLEG): container finished" podID="c9601351-e31d-4fd0-9361-4859651458e8" containerID="7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d" exitCode=0 Nov 03 22:58:46.816195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.816043 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" event={"ID":"c9601351-e31d-4fd0-9361-4859651458e8","Type":"ContainerDied","Data":"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d"} Nov 03 22:58:46.816195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.816096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" event={"ID":"c9601351-e31d-4fd0-9361-4859651458e8","Type":"ContainerDied","Data":"08e104aee45e44b6f4d2e87715130521d16a44020258bf36afc22797c452cc20"} Nov 03 22:58:46.816195 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.816118 5051 scope.go:122] "RemoveContainer" containerID="7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d" Nov 03 22:58:46.816346 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.816222 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq" Nov 03 22:58:46.818699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.818562 5051 generic.go:356] "Generic (PLEG): container finished" podID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerID="15abb30c357316f21974c3b2fa97c98ebc9604d9594e3acadf50a7a03d9c3cfc" exitCode=0 Nov 03 22:58:46.818699 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.818640 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" event={"ID":"740d86ff-0522-4bf4-8ecc-89f0ce1457e7","Type":"ContainerDied","Data":"15abb30c357316f21974c3b2fa97c98ebc9604d9594e3acadf50a7a03d9c3cfc"} Nov 03 22:58:46.830678 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.830651 5051 scope.go:122] "RemoveContainer" containerID="7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d" Nov 03 22:58:46.834000 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:58:46.833947 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d\": container with ID starting with 7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d not found: ID does not exist" containerID="7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d" Nov 03 22:58:46.834126 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.834017 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d"} err="failed to get container status \"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d\": rpc error: code = NotFound desc = could not find container \"7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d\": container with ID starting with 7f67558c797bd1000c493da2ec9d092a2d1e9cb89ad8a98e0e71e7f61d16fc5d not found: ID does not exist" Nov 03 22:58:46.836957 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.836927 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837335 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837307 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.837575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837546 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837597 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837638 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837674 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837759 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.837857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.837843 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.838048 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.838028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.838125 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.838094 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\") pod \"c9601351-e31d-4fd0-9361-4859651458e8\" (UID: \"c9601351-e31d-4fd0-9361-4859651458e8\") " Nov 03 22:58:46.840600 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.840563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.841086 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.841052 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.841400 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.841381 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.841596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.841577 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.841841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.841811 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.849438 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.849410 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.851688 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.851660 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "kube-api-access-zbz4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.853364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.853335 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:46.854150 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.854124 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy" pod "c9601351-e31d-4fd0-9361-4859651458e8" (UID: "c9601351-e31d-4fd0-9361-4859651458e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.918175 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.918150 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:58:46.942505 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942470 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942527 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942589 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942637 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942704 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942691 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942790 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.942938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942812 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.942938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942885 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.942938 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.942916 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.943183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943025 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.943183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.943183 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943151 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.943347 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943238 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\") pod \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\" (UID: \"740d86ff-0522-4bf4-8ecc-89f0ce1457e7\") " Nov 03 22:58:46.943347 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943293 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.943813 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943794 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbz4d\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-kube-api-access-zbz4d\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943821 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9601351-e31d-4fd0-9361-4859651458e8-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943834 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9601351-e31d-4fd0-9361-4859651458e8-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943848 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943861 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943878 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943891 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.943916 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943904 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.944222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943916 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.944222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943931 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.944222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943945 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.944222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943958 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9601351-e31d-4fd0-9361-4859651458e8-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.944222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.943972 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9601351-e31d-4fd0-9361-4859651458e8-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.945854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.945835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.945932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.945853 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.945932 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.945895 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "kube-api-access-czbdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.946140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:46.946120 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo" pod "740d86ff-0522-4bf4-8ecc-89f0ce1457e7" (UID: "740d86ff-0522-4bf4-8ecc-89f0ce1457e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:47.045977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.045890 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czbdf\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-kube-api-access-czbdf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.045977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.045920 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.045977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.045929 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.045977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.045937 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740d86ff-0522-4bf4-8ecc-89f0ce1457e7-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.273389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.273354 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:58:47.287743 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.287708 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-8jkmq"] Nov 03 22:58:47.831093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.831003 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" event={"ID":"740d86ff-0522-4bf4-8ecc-89f0ce1457e7","Type":"ContainerDied","Data":"10c94747f5b2e7b0f2baac7a9010aae9d45aa8fd5886721e09683ec4cefdab3c"} Nov 03 22:58:47.831093 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.831067 5051 scope.go:122] "RemoveContainer" containerID="15abb30c357316f21974c3b2fa97c98ebc9604d9594e3acadf50a7a03d9c3cfc" Nov 03 22:58:47.831644 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:47.830977 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst" Nov 03 22:58:48.006010 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:48.005963 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:58:48.043328 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:48.043293 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-xfkst"] Nov 03 22:58:48.412170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:48.412142 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" path="/var/lib/kubelet/pods/740d86ff-0522-4bf4-8ecc-89f0ce1457e7/volumes" Nov 03 22:58:48.412631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:48.412616 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9601351-e31d-4fd0-9361-4859651458e8" path="/var/lib/kubelet/pods/c9601351-e31d-4fd0-9361-4859651458e8/volumes" Nov 03 22:58:51.625324 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:58:51.625287 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:58:54.386703 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.386660 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8"] Nov 03 22:58:54.387756 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387732 5051 cpu_manager.go:414] "containerMap: removing container" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" Nov 03 22:58:54.387848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387757 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" Nov 03 22:58:54.387848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387800 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" containerName="revoke-keys" Nov 03 22:58:54.387848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387808 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" containerName="revoke-keys" Nov 03 22:58:54.387848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387824 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" Nov 03 22:58:54.387848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.387831 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" Nov 03 22:58:54.388170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.388034 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" containerName="revoke-keys" Nov 03 22:58:54.388170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.388046 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9601351-e31d-4fd0-9361-4859651458e8" containerName="istio-proxy" Nov 03 22:58:54.388170 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.388069 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="740d86ff-0522-4bf4-8ecc-89f0ce1457e7" containerName="istio-proxy" Nov 03 22:58:54.393751 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.393725 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.426692 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.426670 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd"] Nov 03 22:58:54.430841 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.430825 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.445922 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.445888 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8"] Nov 03 22:58:54.446325 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.446306 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:54.471540 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.471512 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:58:54.476216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.476184 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.476347 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.476278 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjmq\" (UniqueName: \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.476450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.476435 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.508281 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.508250 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd"] Nov 03 22:58:54.578020 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.577949 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjmq\" (UniqueName: \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.578222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578102 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.578222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578148 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.578222 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578216 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh72r\" (UniqueName: \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.578530 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578509 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.578634 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578568 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.578946 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.578922 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.581521 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.581498 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.639164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.639097 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjmq\" (UniqueName: \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\") pod \"maas-api-revoke-sub-b3e160a5-hn7x8\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.679578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.679530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.679764 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.679746 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.679835 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.679820 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh72r\" (UniqueName: \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.680565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.680537 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.682214 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.682190 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.721025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.718269 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:54.745052 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.741965 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh72r\" (UniqueName: \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\") pod \"maas-api-revoke-sub-15831477-rbptd\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:54.747117 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:54.746616 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:55.153542 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.153508 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8"] Nov 03 22:58:55.177177 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:58:55.177146 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac22e0f7_f884_4b17_b83c_7045027756a3.slice/crio-daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6 WatchSource:0}: Error finding container daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6: Status 404 returned error can't find the container with id daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6 Nov 03 22:58:55.209604 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:58:55.209578 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f2b4f1_46a3_412b_9c54_27f3512e88b5.slice/crio-fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff WatchSource:0}: Error finding container fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff: Status 404 returned error can't find the container with id fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff Nov 03 22:58:55.210149 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.210126 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd"] Nov 03 22:58:55.798731 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.798694 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 22:58:55.804097 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.804072 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.858772 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.858733 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a9ab23c4-openshift-default-dockercfg-rxq5w\"" Nov 03 22:58:55.867428 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.867395 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 22:58:55.927280 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927301 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927430 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927627 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927563 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927687 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927628 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927748 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927730 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927853 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927830 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927877 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.927978 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.927927 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrjd\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:55.953121 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.953086 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" event={"ID":"ac22e0f7-f884-4b17-b83c-7045027756a3","Type":"ContainerStarted","Data":"7e111857c3c6d3b9ff2c1d8ae508b109294b462eb46e62df6fb769d1b29b0b66"} Nov 03 22:58:55.953301 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.953145 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" event={"ID":"ac22e0f7-f884-4b17-b83c-7045027756a3","Type":"ContainerStarted","Data":"daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6"} Nov 03 22:58:55.954693 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.954670 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" event={"ID":"09f2b4f1-46a3-412b-9c54-27f3512e88b5","Type":"ContainerStarted","Data":"358bd48f7b5e631e325b3f6c7fbe6ea02a0533919e5d9804650acbabba314428"} Nov 03 22:58:55.954805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:55.954709 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" event={"ID":"09f2b4f1-46a3-412b-9c54-27f3512e88b5","Type":"ContainerStarted","Data":"fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff"} Nov 03 22:58:56.029757 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029725 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.029979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029769 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.029979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029803 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrjd\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.029979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029838 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.029979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029866 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.029979 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.029966 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030122 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030188 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030323 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030385 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030288 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030477 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030419 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030610 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030508 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.030671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.030654 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.032735 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.032713 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.032850 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.032834 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.126870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.126754 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrjd\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.138681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.138652 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.250060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.238458 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" podStartSLOduration=2.238445482 podStartE2EDuration="2.238445482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:56.11754803 +0000 UTC m=+2209.854892259" watchObservedRunningTime="2026-11-03 22:58:56.238445482 +0000 UTC m=+2209.975789706" Nov 03 22:58:56.250060 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.240631 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" podStartSLOduration=2.240618584 podStartE2EDuration="2.240618584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:56.237782338 +0000 UTC m=+2209.975126562" watchObservedRunningTime="2026-11-03 22:58:56.240618584 +0000 UTC m=+2209.977962874" Nov 03 22:58:56.420498 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.420413 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:56.652946 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.652906 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:58:56.653164 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.653125 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" containerID="cri-o://0f2f96a4a9023bf9562b8841099178e341f20c133e93fdca8d84fe3352e10a9c" gracePeriod=30 Nov 03 22:58:56.719080 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.718975 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647"] Nov 03 22:58:56.725669 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.725638 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:58:56.725952 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.725893 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" containerID="cri-o://dbf4beb5af35aea8da9ad9e4a8e1b9c490cd78b9a97a568d26817baaf3118d6f" gracePeriod=30 Nov 03 22:58:56.730273 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.730239 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-0406a8-2d647"] Nov 03 22:58:56.880719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.880690 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 22:58:56.883151 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:58:56.883116 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ef9bf7_4ebe_4d1e_85a1_d716756f55ae.slice/crio-93ed9b0fd91a2509756b252656cc0ad1ad0072ac0bb6d8fd384bcfa1d2c86576 WatchSource:0}: Error finding container 93ed9b0fd91a2509756b252656cc0ad1ad0072ac0bb6d8fd384bcfa1d2c86576: Status 404 returned error can't find the container with id 93ed9b0fd91a2509756b252656cc0ad1ad0072ac0bb6d8fd384bcfa1d2c86576 Nov 03 22:58:56.886213 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.886176 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.886324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.886245 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.886324 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.886283 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.969271 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.969210 5051 generic.go:356] "Generic (PLEG): container finished" podID="ac22e0f7-f884-4b17-b83c-7045027756a3" containerID="7e111857c3c6d3b9ff2c1d8ae508b109294b462eb46e62df6fb769d1b29b0b66" exitCode=0 Nov 03 22:58:56.969412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.969323 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" event={"ID":"ac22e0f7-f884-4b17-b83c-7045027756a3","Type":"ContainerDied","Data":"7e111857c3c6d3b9ff2c1d8ae508b109294b462eb46e62df6fb769d1b29b0b66"} Nov 03 22:58:56.971758 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.971727 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" event={"ID":"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae","Type":"ContainerStarted","Data":"93ed9b0fd91a2509756b252656cc0ad1ad0072ac0bb6d8fd384bcfa1d2c86576"} Nov 03 22:58:56.973684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.973654 5051 generic.go:356] "Generic (PLEG): container finished" podID="09f2b4f1-46a3-412b-9c54-27f3512e88b5" containerID="358bd48f7b5e631e325b3f6c7fbe6ea02a0533919e5d9804650acbabba314428" exitCode=0 Nov 03 22:58:56.973821 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:56.973749 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" event={"ID":"09f2b4f1-46a3-412b-9c54-27f3512e88b5","Type":"ContainerDied","Data":"358bd48f7b5e631e325b3f6c7fbe6ea02a0533919e5d9804650acbabba314428"} Nov 03 22:58:58.000012 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:57.997473 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" event={"ID":"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae","Type":"ContainerStarted","Data":"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd"} Nov 03 22:58:58.184019 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.171883 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podStartSLOduration=3.171862367 podStartE2EDuration="3.171862367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:58.170610303 +0000 UTC m=+2211.907954532" watchObservedRunningTime="2026-11-03 22:58:58.171862367 +0000 UTC m=+2211.909206590" Nov 03 22:58:58.309181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.305610 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:58.310849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.310826 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:58.411229 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.411197 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1812bbeb-24b2-44c5-a34e-2289476123fa" path="/var/lib/kubelet/pods/1812bbeb-24b2-44c5-a34e-2289476123fa/volumes" Nov 03 22:58:58.420626 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.420604 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:58:58.422317 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.422297 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:58:58.422405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.422341 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:58:58.499025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.498485 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\" (UniqueName: \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\") pod \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " Nov 03 22:58:58.499025 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.498971 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\") pod \"ac22e0f7-f884-4b17-b83c-7045027756a3\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " Nov 03 22:58:58.499279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.499056 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\") pod \"ac22e0f7-f884-4b17-b83c-7045027756a3\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " Nov 03 22:58:58.499279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.499187 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\") pod \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " Nov 03 22:58:58.499279 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.499223 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\") pod \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\" (UID: \"09f2b4f1-46a3-412b-9c54-27f3512e88b5\") " Nov 03 22:58:58.499434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.499403 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\" (UniqueName: \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\") pod \"ac22e0f7-f884-4b17-b83c-7045027756a3\" (UID: \"ac22e0f7-f884-4b17-b83c-7045027756a3\") " Nov 03 22:58:58.499598 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.499568 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca" pod "ac22e0f7-f884-4b17-b83c-7045027756a3" (UID: "ac22e0f7-f884-4b17-b83c-7045027756a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:58.500186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.500158 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:58.500186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.500165 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca" pod "09f2b4f1-46a3-412b-9c54-27f3512e88b5" (UID: "09f2b4f1-46a3-412b-9c54-27f3512e88b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:58.519376 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.519345 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r" pod "09f2b4f1-46a3-412b-9c54-27f3512e88b5" (UID: "09f2b4f1-46a3-412b-9c54-27f3512e88b5"). InnerVolumeSpecName "kube-api-access-rh72r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:58.520148 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.520121 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca" pod "09f2b4f1-46a3-412b-9c54-27f3512e88b5" (UID: "09f2b4f1-46a3-412b-9c54-27f3512e88b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:58.521520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.521495 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq" pod "ac22e0f7-f884-4b17-b83c-7045027756a3" (UID: "ac22e0f7-f884-4b17-b83c-7045027756a3"). InnerVolumeSpecName "kube-api-access-hwjmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:58.522727 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.522704 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca" pod "ac22e0f7-f884-4b17-b83c-7045027756a3" (UID: "ac22e0f7-f884-4b17-b83c-7045027756a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:58.603919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.603879 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjmq\" (UniqueName: \"kubernetes.io/projected/ac22e0f7-f884-4b17-b83c-7045027756a3-kube-api-access-hwjmq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:58.603919 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.603918 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh72r\" (UniqueName: \"kubernetes.io/projected/09f2b4f1-46a3-412b-9c54-27f3512e88b5-kube-api-access-rh72r\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:58.604218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.603933 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac22e0f7-f884-4b17-b83c-7045027756a3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:58.604218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.603948 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:58.604218 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:58.603961 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f2b4f1-46a3-412b-9c54-27f3512e88b5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:59.010044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.010012 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" Nov 03 22:58:59.010044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.010032 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd" event={"ID":"09f2b4f1-46a3-412b-9c54-27f3512e88b5","Type":"ContainerDied","Data":"fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff"} Nov 03 22:58:59.010539 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.010091 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4087b070d544fd6d839d7242f77a6feb529761db7bd70e5fa57ec577a0d7ff" Nov 03 22:58:59.011909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.011893 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" Nov 03 22:58:59.011909 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.011896 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8" event={"ID":"ac22e0f7-f884-4b17-b83c-7045027756a3","Type":"ContainerDied","Data":"daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6"} Nov 03 22:58:59.012085 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.011934 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa2491ecd7acd95ba18d49baf25a0f7b4cdedbe70697e9d338583255ccbf3d6" Nov 03 22:58:59.438512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.421591 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:58:59.438512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.421672 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:58:59.442867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.442749 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:59.442867 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:58:59.442802 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:59:00.421877 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:00.421843 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:59:00.422315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:00.421895 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:59:01.421092 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:01.421059 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:59:01.421293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:01.421113 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:59:02.039159 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.039130 5051 generic.go:356] "Generic (PLEG): container finished" podID="82178681-caca-4f12-a0ad-a657a9277729" containerID="dbf4beb5af35aea8da9ad9e4a8e1b9c490cd78b9a97a568d26817baaf3118d6f" exitCode=0 Nov 03 22:59:02.039557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.039215 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" event={"ID":"82178681-caca-4f12-a0ad-a657a9277729","Type":"ContainerDied","Data":"dbf4beb5af35aea8da9ad9e4a8e1b9c490cd78b9a97a568d26817baaf3118d6f"} Nov 03 22:59:02.039557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.039279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" event={"ID":"82178681-caca-4f12-a0ad-a657a9277729","Type":"ContainerDied","Data":"5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473"} Nov 03 22:59:02.039557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.039307 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cca7df890df93e5950e7db91c44e53495731a612b7bbeb66981c0178f8b2473" Nov 03 22:59:02.041137 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.041114 5051 generic.go:356] "Generic (PLEG): container finished" podID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerID="0f2f96a4a9023bf9562b8841099178e341f20c133e93fdca8d84fe3352e10a9c" exitCode=0 Nov 03 22:59:02.041248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.041201 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" event={"ID":"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c","Type":"ContainerDied","Data":"0f2f96a4a9023bf9562b8841099178e341f20c133e93fdca8d84fe3352e10a9c"} Nov 03 22:59:02.048176 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.048157 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:59:02.124294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124262 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124467 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124302 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124520 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124499 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124565 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124546 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124580 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124639 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124621 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124660 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124760 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124713 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.124870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124751 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.124870 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:02.124970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.124896 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\") pod \"82178681-caca-4f12-a0ad-a657a9277729\" (UID: \"82178681-caca-4f12-a0ad-a657a9277729\") " Nov 03 22:59:02.125071 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125039 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.125127 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125066 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.125199 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125136 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.125861 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125838 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.125983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125864 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.125983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125879 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.125983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125890 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.125983 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.125902 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82178681-caca-4f12-a0ad-a657a9277729-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.127451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.127428 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:02.127560 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.127456 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "kube-api-access-tgs4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:02.127653 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.127634 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:02.127796 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.127777 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy" pod "82178681-caca-4f12-a0ad-a657a9277729" (UID: "82178681-caca-4f12-a0ad-a657a9277729"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.246013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.245038 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgs4c\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-kube-api-access-tgs4c\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.246013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.245071 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82178681-caca-4f12-a0ad-a657a9277729-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.246013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.245085 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82178681-caca-4f12-a0ad-a657a9277729-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.246013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.245096 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82178681-caca-4f12-a0ad-a657a9277729-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.420851 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.420817 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:59:02.421140 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.420866 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:59:02.833423 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.833400 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:59:02.927102 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927070 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927119 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927152 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927175 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927195 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927303 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927262 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927379 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927421 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927577 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927526 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.927737 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927653 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\") pod \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\" (UID: \"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c\") " Nov 03 22:59:02.927822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.927780 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.928276 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928105 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.928276 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928175 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:02.928276 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928233 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:02.928516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928349 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.928516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928371 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.928516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928386 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.928516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928398 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.928516 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.928411 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:02.930304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.930261 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:02.930718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.930691 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:02.930801 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.930748 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "kube-api-access-gdlgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:02.930860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:02.930804 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy" pod "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" (UID: "c8de30bd-5149-4e72-a5fd-1cdb838bbe2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:03.031771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.031737 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlgq\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-kube-api-access-gdlgq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:03.031771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.031769 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:03.031771 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.031778 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:03.032181 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.031791 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:03.097389 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.097303 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" Nov 03 22:59:03.098057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.097385 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc" Nov 03 22:59:03.098243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.097427 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl" event={"ID":"c8de30bd-5149-4e72-a5fd-1cdb838bbe2c","Type":"ContainerDied","Data":"a6db5d342e9f73c704308a44e4adc9037be1adc6d8f55ccf14e4b87ca40589d9"} Nov 03 22:59:03.098546 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.098533 5051 scope.go:122] "RemoveContainer" containerID="0f2f96a4a9023bf9562b8841099178e341f20c133e93fdca8d84fe3352e10a9c" Nov 03 22:59:03.288823 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.288786 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:59:03.293174 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.293145 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-rsqdc"] Nov 03 22:59:03.416821 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.415050 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:59:03.422575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.422219 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-59hxl"] Nov 03 22:59:03.422575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.422253 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 22:59:03.422575 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:03.422356 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 22:59:04.409892 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:04.409809 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82178681-caca-4f12-a0ad-a657a9277729" path="/var/lib/kubelet/pods/82178681-caca-4f12-a0ad-a657a9277729/volumes" Nov 03 22:59:04.410590 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:04.410571 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" path="/var/lib/kubelet/pods/c8de30bd-5149-4e72-a5fd-1cdb838bbe2c/volumes" Nov 03 22:59:04.426057 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:04.426025 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:59:04.426315 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:04.426295 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:59:04.426930 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:04.426913 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 22:59:08.633584 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:08.633549 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb"] Nov 03 22:59:08.691177 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:08.691137 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-aacab9-4ddhb"] Nov 03 22:59:10.411828 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.411791 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e727ae-5c2e-40f6-bfff-e4d5c7967140" path="/var/lib/kubelet/pods/09e727ae-5c2e-40f6-bfff-e4d5c7967140/volumes" Nov 03 22:59:10.676059 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.675965 5051 scope.go:122] "RemoveContainer" containerID="88a57f06b4851aa05950aec906846042df65d2641a714b42074cbda07f269708" Nov 03 22:59:10.687311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.687285 5051 scope.go:122] "RemoveContainer" containerID="dbf4beb5af35aea8da9ad9e4a8e1b9c490cd78b9a97a568d26817baaf3118d6f" Nov 03 22:59:10.955671 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.955591 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 22:59:10.956381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956363 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956383 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956419 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ac22e0f7-f884-4b17-b83c-7045027756a3" containerName="revoke-keys" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956424 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac22e0f7-f884-4b17-b83c-7045027756a3" containerName="revoke-keys" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956456 5051 cpu_manager.go:414] "containerMap: removing container" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956461 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956469 5051 cpu_manager.go:414] "containerMap: removing container" podUID="09f2b4f1-46a3-412b-9c54-27f3512e88b5" containerName="revoke-keys" Nov 03 22:59:10.956471 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956474 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f2b4f1-46a3-412b-9c54-27f3512e88b5" containerName="revoke-keys" Nov 03 22:59:10.956710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956607 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac22e0f7-f884-4b17-b83c-7045027756a3" containerName="revoke-keys" Nov 03 22:59:10.956710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956631 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="82178681-caca-4f12-a0ad-a657a9277729" containerName="istio-proxy" Nov 03 22:59:10.956710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956642 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f2b4f1-46a3-412b-9c54-27f3512e88b5" containerName="revoke-keys" Nov 03 22:59:10.956710 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.956653 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8de30bd-5149-4e72-a5fd-1cdb838bbe2c" containerName="istio-proxy" Nov 03 22:59:10.962008 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:10.961970 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021572 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021633 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021654 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021781 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021773 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021854 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021872 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.021927 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021898 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.022109 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.021978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv49b\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.022178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.022156 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123480 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123573 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123621 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123644 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123723 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123816 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123843 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.123935 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123884 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.124118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.123935 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv49b\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.124118 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.124081 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.124243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.124123 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.124243 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.124214 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.124451 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.124432 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.125158 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.125133 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.126538 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.126506 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.126737 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.126717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.177557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.177520 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 22:59:11.188216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.185664 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv49b\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.188216 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.186791 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.279889 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.279796 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:11.755774 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:59:11.755741 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab58e595_3a1d_40d5_89b3_a965360f43f9.slice/crio-382a4201def5e13b11fd5975c234718b790e8943bb39d6dd343dccd1becd6db4 WatchSource:0}: Error finding container 382a4201def5e13b11fd5975c234718b790e8943bb39d6dd343dccd1becd6db4: Status 404 returned error can't find the container with id 382a4201def5e13b11fd5975c234718b790e8943bb39d6dd343dccd1becd6db4 Nov 03 22:59:11.756368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.756342 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 22:59:11.759708 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.759659 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:11.759831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.759755 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:11.759831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:11.759805 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:12.229251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.229213 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" event={"ID":"ab58e595-3a1d-40d5-89b3-a965360f43f9","Type":"ContainerStarted","Data":"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39"} Nov 03 22:59:12.229445 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.229278 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" event={"ID":"ab58e595-3a1d-40d5-89b3-a965360f43f9","Type":"ContainerStarted","Data":"382a4201def5e13b11fd5975c234718b790e8943bb39d6dd343dccd1becd6db4"} Nov 03 22:59:12.280496 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.280464 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:12.282469 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.282436 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:12.282596 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.282511 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:12.446531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:12.446476 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podStartSLOduration=2.446462065 podStartE2EDuration="2.446462065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:12.443683619 +0000 UTC m=+2226.181027843" watchObservedRunningTime="2026-11-03 22:59:12.446462065 +0000 UTC m=+2226.183806290" Nov 03 22:59:13.239862 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.239831 5051 generic.go:356] "Generic (PLEG): container finished" podID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerID="255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27" exitCode=0 Nov 03 22:59:13.240474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.239910 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerDied","Data":"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27"} Nov 03 22:59:13.240631 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.240614 5051 scope.go:122] "RemoveContainer" containerID="255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27" Nov 03 22:59:13.242271 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.242241 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:13.242368 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.242310 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:13.242409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.242365 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:13.284013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.283401 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:13.284013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:13.283461 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:14.250860 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.250817 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerStarted","Data":"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b"} Nov 03 22:59:14.281450 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.281408 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:14.281777 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.281743 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:14.511519 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.511432 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:59:14.512713 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.512682 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:14.512843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:14.512753 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:15.338832 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:15.338787 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:15.339373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:15.338855 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:15.500854 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:15.500817 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:15.501219 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:15.501187 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:16.287016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.281107 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:16.287016 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.281183 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:16.501385 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.501345 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:16.502034 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.501982 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:16.534138 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.534104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:59:16.732345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.732309 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:59:16.742188 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.742162 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:59:16.743294 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.743267 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" containerID="cri-o://70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069" gracePeriod=30 Nov 03 22:59:16.743609 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:16.742568 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" containerID="cri-o://a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3" gracePeriod=30 Nov 03 22:59:17.281017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:17.280945 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:17.281244 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:17.281036 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:17.504198 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:17.504146 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:17.504844 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:17.504796 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:18.373115 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:18.373079 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 22:59:18.373418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:18.373387 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 22:59:18.499037 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:18.498979 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:18.499351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:18.499319 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:19.296941 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:19.296905 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:19.298123 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:19.298104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:19.298970 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:19.298946 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 22:59:19.499030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:19.498482 5051 patch_prober.go:29] interesting pod/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 22:59:19.499030 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:19.498550 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 22:59:20.503373 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:20.503340 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:59:20.504719 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:20.504697 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 22:59:21.626094 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:21.626058 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 22:59:22.057405 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.057375 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:59:22.099578 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.099555 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:59:22.164178 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164144 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164257 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164289 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164322 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164374 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164525 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164501 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164585 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164691 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164625 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.164744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164705 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164805 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164757 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\") pod \"c84c9481-15ed-4c60-9f7b-697adb1e2506\" (UID: \"c84c9481-15ed-4c60-9f7b-697adb1e2506\") " Nov 03 22:59:22.164974 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.164949 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:22.165462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.165434 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.165557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.165468 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84c9481-15ed-4c60-9f7b-697adb1e2506-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.165750 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.165691 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.165893 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.165757 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.166302 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.166281 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.167299 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.167279 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.167409 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.167340 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "kube-api-access-7rdjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.167696 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.167677 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:22.168144 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.168127 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy" pod "c84c9481-15ed-4c60-9f7b-697adb1e2506" (UID: "c84c9481-15ed-4c60-9f7b-697adb1e2506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.266746 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266656 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.266928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266752 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.266928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266796 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.266928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266824 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.266928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266859 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.266928 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.267345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.266970 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.267345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267071 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.267345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267167 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.267345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267191 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.267345 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\") pod \"ba9108f4-a96b-413c-a8a8-9711644bb826\" (UID: \"ba9108f4-a96b-413c-a8a8-9711644bb826\") " Nov 03 22:59:22.267617 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267452 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:22.267984 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.267830 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.268130 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268115 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268137 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268155 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268170 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9108f4-a96b-413c-a8a8-9711644bb826-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268196 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268184 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268196 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268210 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268225 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268237 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268249 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdjq\" (UniqueName: \"kubernetes.io/projected/c84c9481-15ed-4c60-9f7b-697adb1e2506-kube-api-access-7rdjq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.268418 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.268263 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.269069 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.269011 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.269896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.269869 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.270412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.270385 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "kube-api-access-5xpfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.270605 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.270583 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.271973 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.271954 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo" pod "ba9108f4-a96b-413c-a8a8-9711644bb826" (UID: "ba9108f4-a96b-413c-a8a8-9711644bb826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:22.369569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.369532 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.369569 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.369568 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpfl\" (UniqueName: \"kubernetes.io/projected/ba9108f4-a96b-413c-a8a8-9711644bb826-kube-api-access-5xpfl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.369818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.369586 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.369818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.369599 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.369818 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.369613 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9108f4-a96b-413c-a8a8-9711644bb826-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.566227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.566135 5051 generic.go:356] "Generic (PLEG): container finished" podID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerID="a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3" exitCode=0 Nov 03 22:59:22.566227 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.566198 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" Nov 03 22:59:22.566474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.566226 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" event={"ID":"ba9108f4-a96b-413c-a8a8-9711644bb826","Type":"ContainerDied","Data":"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3"} Nov 03 22:59:22.566474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.566284 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9" event={"ID":"ba9108f4-a96b-413c-a8a8-9711644bb826","Type":"ContainerDied","Data":"2c478e86c1e4cc49aa3733fc5247e53d348a68704737c7f5c3bcf144474d0633"} Nov 03 22:59:22.566474 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.566315 5051 scope.go:122] "RemoveContainer" containerID="a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3" Nov 03 22:59:22.568146 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.568123 5051 generic.go:356] "Generic (PLEG): container finished" podID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerID="70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069" exitCode=0 Nov 03 22:59:22.568284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.568181 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" event={"ID":"c84c9481-15ed-4c60-9f7b-697adb1e2506","Type":"ContainerDied","Data":"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069"} Nov 03 22:59:22.568284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.568208 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" Nov 03 22:59:22.568284 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.568219 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm" event={"ID":"c84c9481-15ed-4c60-9f7b-697adb1e2506","Type":"ContainerDied","Data":"786c13716e238b028419629aa20eebb982cfd3ff61d22fee4ecdfdd383432c02"} Nov 03 22:59:22.580385 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.580366 5051 scope.go:122] "RemoveContainer" containerID="a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3" Nov 03 22:59:22.580713 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:22.580684 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3\": container with ID starting with a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3 not found: ID does not exist" containerID="a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3" Nov 03 22:59:22.580802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.580717 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3"} err="failed to get container status \"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3\": rpc error: code = NotFound desc = could not find container \"a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3\": container with ID starting with a3a140efdddb31e6bce4dbbf5dfbfe61247cc67867dc280d9f5183a201741ed3 not found: ID does not exist" Nov 03 22:59:22.580802 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.580740 5051 scope.go:122] "RemoveContainer" containerID="70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069" Nov 03 22:59:22.594248 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.594224 5051 scope.go:122] "RemoveContainer" containerID="70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069" Nov 03 22:59:22.594591 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:22.594568 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069\": container with ID starting with 70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069 not found: ID does not exist" containerID="70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069" Nov 03 22:59:22.594673 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.594611 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069"} err="failed to get container status \"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069\": rpc error: code = NotFound desc = could not find container \"70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069\": container with ID starting with 70f8fdcd6ad3a054d318a474a9c5e2143ff77e271d2b595e3483b500b8594069 not found: ID does not exist" Nov 03 22:59:22.737365 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.737332 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:59:22.786945 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.786761 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-7xpv9"] Nov 03 22:59:22.899849 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.899807 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:59:22.906266 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:22.906236 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-zktwm"] Nov 03 22:59:24.411843 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:24.411804 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" path="/var/lib/kubelet/pods/ba9108f4-a96b-413c-a8a8-9711644bb826/volumes" Nov 03 22:59:24.412527 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:24.412505 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" path="/var/lib/kubelet/pods/c84c9481-15ed-4c60-9f7b-697adb1e2506/volumes" Nov 03 22:59:26.057573 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.057534 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 22:59:26.058272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058256 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" Nov 03 22:59:26.058272 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058273 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" Nov 03 22:59:26.058391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058291 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" Nov 03 22:59:26.058391 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058296 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" Nov 03 22:59:26.058494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058391 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9108f4-a96b-413c-a8a8-9711644bb826" containerName="istio-proxy" Nov 03 22:59:26.058494 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.058399 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84c9481-15ed-4c60-9f7b-697adb1e2506" containerName="istio-proxy" Nov 03 22:59:26.063827 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.063809 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.145766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.145738 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.145934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.145784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.145934 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.145914 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.145944 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146114 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.145965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.146114 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.146193 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146311 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.146248 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgszl\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.146348 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.146322 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.165848 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.165816 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 22:59:26.247208 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247242 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247291 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247403 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247387 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgszl\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247433 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247610 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247730 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247679 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247780 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.247899 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.247836 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.248044 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.248011 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.248112 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.248077 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.248557 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.248535 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.249822 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.249803 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.250226 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.250210 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.308246 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.308176 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.308364 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.308345 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgszl\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.379910 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.379883 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:26.822136 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.822107 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 22:59:26.824700 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:59:26.824671 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb71b56_1aac_4e65_abb9_a52f7468225b.slice/crio-8260c19229cf1b7ee2ee0630f2f66eb82e050d6dca9980489805f54c8e39fa17 WatchSource:0}: Error finding container 8260c19229cf1b7ee2ee0630f2f66eb82e050d6dca9980489805f54c8e39fa17: Status 404 returned error can't find the container with id 8260c19229cf1b7ee2ee0630f2f66eb82e050d6dca9980489805f54c8e39fa17 Nov 03 22:59:26.827500 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.827467 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:26.827579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.827533 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:26.827579 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:26.827560 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:27.651490 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:27.651439 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" event={"ID":"afb71b56-1aac-4e65-abb9-a52f7468225b","Type":"ContainerStarted","Data":"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243"} Nov 03 22:59:27.651913 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:27.651502 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" event={"ID":"afb71b56-1aac-4e65-abb9-a52f7468225b","Type":"ContainerStarted","Data":"8260c19229cf1b7ee2ee0630f2f66eb82e050d6dca9980489805f54c8e39fa17"} Nov 03 22:59:27.777014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:27.776920 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podStartSLOduration=1.7769010280000002 podStartE2EDuration="1.776901028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:27.774493524 +0000 UTC m=+2241.511837750" watchObservedRunningTime="2026-11-03 22:59:27.776901028 +0000 UTC m=+2241.514245281" Nov 03 22:59:28.387131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:28.383390 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:28.387131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:28.387091 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 22:59:28.387437 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:28.387141 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 22:59:29.383786 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:29.383748 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 22:59:29.384412 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:29.384381 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 22:59:30.381544 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:30.381501 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 22:59:30.381744 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:30.381580 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 22:59:31.381017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.380429 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 22:59:31.381017 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.380512 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 22:59:31.837977 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.837899 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl"] Nov 03 22:59:31.842512 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.842487 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:31.899773 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.899745 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl"] Nov 03 22:59:31.943689 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.943654 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:31.943857 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.943739 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:59:31.983261 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.983225 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:31.983462 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.983369 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwk52\" (UniqueName: \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:31.983556 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:31.983532 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.085035 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.084971 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.085251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.085089 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwk52\" (UniqueName: \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.085251 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.085202 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.085834 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.085792 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.088217 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.088162 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.205015 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.200486 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwk52\" (UniqueName: \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\") pod \"maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.386014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.382440 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 22:59:32.386014 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.382501 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 22:59:32.484517 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.484485 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:32.887896 ip-10-0-135-87 kubenswrapper[5051]: W1103 22:59:32.887861 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246c92b7_3a24_482b_bc72_0f6e321fd741.slice/crio-22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d WatchSource:0}: Error finding container 22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d: Status 404 returned error can't find the container with id 22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d Nov 03 22:59:32.887896 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:32.887867 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl"] Nov 03 22:59:33.572690 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:33.572596 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:33.573646 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:33.573536 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:33.574468 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:33.574450 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 22:59:33.729186 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:33.729146 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" event={"ID":"246c92b7-3a24-482b-bc72-0f6e321fd741","Type":"ContainerStarted","Data":"248d7e05846a4c5065f368d2e352bb1a8642e8872c9d54054398fe672bec1e03"} Nov 03 22:59:33.729363 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:33.729212 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" event={"ID":"246c92b7-3a24-482b-bc72-0f6e321fd741","Type":"ContainerStarted","Data":"22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d"} Nov 03 22:59:34.893049 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:34.891647 5051 generic.go:356] "Generic (PLEG): container finished" podID="246c92b7-3a24-482b-bc72-0f6e321fd741" containerID="248d7e05846a4c5065f368d2e352bb1a8642e8872c9d54054398fe672bec1e03" exitCode=0 Nov 03 22:59:34.893550 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:34.893112 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" event={"ID":"246c92b7-3a24-482b-bc72-0f6e321fd741","Type":"ContainerDied","Data":"248d7e05846a4c5065f368d2e352bb1a8642e8872c9d54054398fe672bec1e03"} Nov 03 22:59:36.119920 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.119889 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:36.229434 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.229339 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\" (UniqueName: \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\") pod \"246c92b7-3a24-482b-bc72-0f6e321fd741\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " Nov 03 22:59:36.229622 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.229580 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\") pod \"246c92b7-3a24-482b-bc72-0f6e321fd741\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " Nov 03 22:59:36.229695 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.229676 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\") pod \"246c92b7-3a24-482b-bc72-0f6e321fd741\" (UID: \"246c92b7-3a24-482b-bc72-0f6e321fd741\") " Nov 03 22:59:36.230013 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.229972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca" pod "246c92b7-3a24-482b-bc72-0f6e321fd741" (UID: "246c92b7-3a24-482b-bc72-0f6e321fd741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:36.230293 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.230241 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.231684 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.231662 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52" pod "246c92b7-3a24-482b-bc72-0f6e321fd741" (UID: "246c92b7-3a24-482b-bc72-0f6e321fd741"). InnerVolumeSpecName "kube-api-access-vwk52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:36.231825 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.231800 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca" pod "246c92b7-3a24-482b-bc72-0f6e321fd741" (UID: "246c92b7-3a24-482b-bc72-0f6e321fd741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:36.331618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.331571 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246c92b7-3a24-482b-bc72-0f6e321fd741-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.331618 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.331611 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwk52\" (UniqueName: \"kubernetes.io/projected/246c92b7-3a24-482b-bc72-0f6e321fd741-kube-api-access-vwk52\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:36.909470 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.909433 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" Nov 03 22:59:36.909681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.909436 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl" event={"ID":"246c92b7-3a24-482b-bc72-0f6e321fd741","Type":"ContainerDied","Data":"22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d"} Nov 03 22:59:36.909681 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:36.909560 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c4c8ce48d19cc9b5b60f7ad83c01d6e4082e8e6948f2904d19f1617d2a078d" Nov 03 22:59:43.574029 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:43.573973 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:59:43.683381 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:43.683346 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:59:48.575024 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.574943 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" podUID="d891fb36-0e67-4dbf-a1b4-b6228912d368" containerName="payload-processing" containerID="cri-o://939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b" gracePeriod=25 Nov 03 22:59:48.684447 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.684406 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" podUID="8269bf59-fe87-4e27-a265-5e377cd60aa2" containerName="payload-pre-processing" containerID="cri-o://908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017" gracePeriod=25 Nov 03 22:59:48.893111 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.893081 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:59:48.945831 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.945634 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\") pod \"d891fb36-0e67-4dbf-a1b4-b6228912d368\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " Nov 03 22:59:48.946157 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.946123 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume" pod "d891fb36-0e67-4dbf-a1b4-b6228912d368" (UID: "d891fb36-0e67-4dbf-a1b4-b6228912d368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:48.946304 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.946159 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\" (UniqueName: \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\") pod \"d891fb36-0e67-4dbf-a1b4-b6228912d368\" (UID: \"d891fb36-0e67-4dbf-a1b4-b6228912d368\") " Nov 03 22:59:48.947106 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.946919 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891fb36-0e67-4dbf-a1b4-b6228912d368-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:48.949242 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:48.949215 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb" pod "d891fb36-0e67-4dbf-a1b4-b6228912d368" (UID: "d891fb36-0e67-4dbf-a1b4-b6228912d368"). InnerVolumeSpecName "kube-api-access-k5hbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:49.006887 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.006863 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:59:49.047965 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.047929 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5hbb\" (UniqueName: \"kubernetes.io/projected/d891fb36-0e67-4dbf-a1b4-b6228912d368-kube-api-access-k5hbb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:49.065551 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.065523 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s"] Nov 03 22:59:49.070718 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.070688 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2406dd7-cs96s"] Nov 03 22:59:49.096351 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.096317 5051 generic.go:356] "Generic (PLEG): container finished" podID="d891fb36-0e67-4dbf-a1b4-b6228912d368" containerID="939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b" exitCode=0 Nov 03 22:59:49.096531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.096388 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" Nov 03 22:59:49.096531 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.096381 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" event={"ID":"d891fb36-0e67-4dbf-a1b4-b6228912d368","Type":"ContainerDied","Data":"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b"} Nov 03 22:59:49.096667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.096555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6" event={"ID":"d891fb36-0e67-4dbf-a1b4-b6228912d368","Type":"ContainerDied","Data":"a7412fc226991fc10d07d992dcc0cc040cb6debdcd8f3d9ef6b55318a0bc8717"} Nov 03 22:59:49.096667 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.096590 5051 scope.go:122] "RemoveContainer" containerID="939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b" Nov 03 22:59:49.101522 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.101499 5051 generic.go:356] "Generic (PLEG): container finished" podID="8269bf59-fe87-4e27-a265-5e377cd60aa2" containerID="908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017" exitCode=0 Nov 03 22:59:49.101650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.101573 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" event={"ID":"8269bf59-fe87-4e27-a265-5e377cd60aa2","Type":"ContainerDied","Data":"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017"} Nov 03 22:59:49.101650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.101613 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" event={"ID":"8269bf59-fe87-4e27-a265-5e377cd60aa2","Type":"ContainerDied","Data":"94f3742748134ed30f2a14bfec7d0566afa79c9423d8209a14a0101f8e679aba"} Nov 03 22:59:49.101755 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.101659 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f" Nov 03 22:59:49.109650 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.109631 5051 scope.go:122] "RemoveContainer" containerID="939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b" Nov 03 22:59:49.109968 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:49.109944 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b\": container with ID starting with 939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b not found: ID does not exist" containerID="939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b" Nov 03 22:59:49.110089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.109977 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b"} err="failed to get container status \"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b\": rpc error: code = NotFound desc = could not find container \"939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b\": container with ID starting with 939f175b41f6a3f0d521fbd2d6f316fa22ecff6679babaca99d250e97bc48a4b not found: ID does not exist" Nov 03 22:59:49.110089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.110015 5051 scope.go:122] "RemoveContainer" containerID="908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017" Nov 03 22:59:49.127374 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.127355 5051 scope.go:122] "RemoveContainer" containerID="908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017" Nov 03 22:59:49.127718 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:49.127682 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017\": container with ID starting with 908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017 not found: ID does not exist" containerID="908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017" Nov 03 22:59:49.127778 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.127725 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017"} err="failed to get container status \"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017\": rpc error: code = NotFound desc = could not find container \"908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017\": container with ID starting with 908b8cd42cfe22d19e8c3ff7d68070e12315afe7ecd8e4f341131809df14c017 not found: ID does not exist" Nov 03 22:59:49.148720 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.148696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\") pod \"8269bf59-fe87-4e27-a265-5e377cd60aa2\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " Nov 03 22:59:49.148879 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.148805 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\" (UniqueName: \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\") pod \"8269bf59-fe87-4e27-a265-5e377cd60aa2\" (UID: \"8269bf59-fe87-4e27-a265-5e377cd60aa2\") " Nov 03 22:59:49.149089 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.149067 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume" pod "8269bf59-fe87-4e27-a265-5e377cd60aa2" (UID: "8269bf59-fe87-4e27-a265-5e377cd60aa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:49.149523 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.149505 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269bf59-fe87-4e27-a265-5e377cd60aa2-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:49.151131 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.151108 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc" pod "8269bf59-fe87-4e27-a265-5e377cd60aa2" (UID: "8269bf59-fe87-4e27-a265-5e377cd60aa2"). InnerVolumeSpecName "kube-api-access-tnbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:49.254033 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.251417 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbwc\" (UniqueName: \"kubernetes.io/projected/8269bf59-fe87-4e27-a265-5e377cd60aa2-kube-api-access-tnbwc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:49.254628 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.254602 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:59:49.258732 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.258706 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a9ab23c4-845f6c8c86-7f2l6"] Nov 03 22:59:49.529766 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.529715 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:59:49.533951 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:49.533927 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a9ab23c4-7f6b8568c9-s4t6f"] Nov 03 22:59:50.429406 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:50.429371 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8269bf59-fe87-4e27-a265-5e377cd60aa2" path="/var/lib/kubelet/pods/8269bf59-fe87-4e27-a265-5e377cd60aa2/volumes" Nov 03 22:59:50.430128 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:50.430105 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ff531e-f3c0-4004-a576-379b696e28fa" path="/var/lib/kubelet/pods/97ff531e-f3c0-4004-a576-379b696e28fa/volumes" Nov 03 22:59:50.430655 ip-10-0-135-87 kubenswrapper[5051]: I1103 22:59:50.430638 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d891fb36-0e67-4dbf-a1b4-b6228912d368" path="/var/lib/kubelet/pods/d891fb36-0e67-4dbf-a1b4-b6228912d368/volumes" Nov 03 22:59:51.624749 ip-10-0-135-87 kubenswrapper[5051]: E1103 22:59:51.624712 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:00:00.183220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.183182 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:00.184166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184142 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8269bf59-fe87-4e27-a265-5e377cd60aa2" containerName="payload-pre-processing" Nov 03 23:00:00.184166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184166 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8269bf59-fe87-4e27-a265-5e377cd60aa2" containerName="payload-pre-processing" Nov 03 23:00:00.184363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184200 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d891fb36-0e67-4dbf-a1b4-b6228912d368" containerName="payload-processing" Nov 03 23:00:00.184363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184208 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d891fb36-0e67-4dbf-a1b4-b6228912d368" containerName="payload-processing" Nov 03 23:00:00.184363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184223 5051 cpu_manager.go:414] "containerMap: removing container" podUID="246c92b7-3a24-482b-bc72-0f6e321fd741" containerName="revoke-keys" Nov 03 23:00:00.184363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184230 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246c92b7-3a24-482b-bc72-0f6e321fd741" containerName="revoke-keys" Nov 03 23:00:00.184572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184396 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8269bf59-fe87-4e27-a265-5e377cd60aa2" containerName="payload-pre-processing" Nov 03 23:00:00.184572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184418 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d891fb36-0e67-4dbf-a1b4-b6228912d368" containerName="payload-processing" Nov 03 23:00:00.184572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.184429 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="246c92b7-3a24-482b-bc72-0f6e321fd741" containerName="revoke-keys" Nov 03 23:00:00.188162 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.188138 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.190127 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.190094 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:00.195881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.195863 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.196652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.196610 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:00.201748 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.201728 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.232674 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.232649 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:00.236983 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.236960 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.239472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.239451 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:00.244945 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.244786 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.257117 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.257092 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:00.261893 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.261866 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9wt\" (UniqueName: \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.262083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.262063 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.262304 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.262282 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.262688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.262666 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.263046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.263020 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ssz\" (UniqueName: \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.263164 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.263081 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q22c\" (UniqueName: \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.266301 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.266279 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:00.267220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.266505 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.270821 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.270803 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:00.271081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.271062 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.275028 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.275008 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:00:00.275349 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.275332 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.279748 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.279727 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:00.284182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.284164 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.284334 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.284263 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:00.290332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.290311 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:00.290439 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.290423 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.293791 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.293774 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:00.299967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.299948 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:00.340343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.340071 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:00.346591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.346561 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:00.365010 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.364946 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.365180 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365025 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.365180 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365104 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.365180 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365155 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.365388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365294 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whd6p\" (UniqueName: \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.365388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365341 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwxt\" (UniqueName: \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.365516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365453 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2hh\" (UniqueName: \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.365580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365524 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.365580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365567 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ssz\" (UniqueName: \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.365701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q22c\" (UniqueName: \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.365760 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365700 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.365760 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365737 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9wt\" (UniqueName: \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.365880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365777 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvblt\" (UniqueName: \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.365880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365813 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.365880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365833 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.366100 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.365901 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbzx\" (UniqueName: \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.366246 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366217 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.366392 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366371 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.366466 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366404 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.366466 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366433 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsh6\" (UniqueName: \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.366466 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366461 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczhj\" (UniqueName: \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.366663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.366634 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.367083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.367064 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.385857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.385445 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:00.390151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.390122 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:00.403112 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.398619 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:00.424216 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.424179 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:00.430411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.430386 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:00.435896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.435831 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ssz\" (UniqueName: \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\") pod \"maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.442980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.442926 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9wt\" (UniqueName: \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\") pod \"maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.444475 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.444445 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q22c\" (UniqueName: \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.468324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468292 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.468477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468332 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.468519 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468471 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.468659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468639 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whd6p\" (UniqueName: \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.468740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468685 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwxt\" (UniqueName: \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.468740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468730 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2hh\" (UniqueName: \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.468858 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468780 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.468923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468885 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.468980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.468933 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvblt\" (UniqueName: \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.469077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469054 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbzx\" (UniqueName: \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.469145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469089 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.469145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469124 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.469145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469134 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.469320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469237 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.469320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsh6\" (UniqueName: \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.469320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469302 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczhj\" (UniqueName: \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.469579 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.469919 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.469820 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.470072 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.470044 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.470155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.470099 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.470155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.470106 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.510737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.510712 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:00.521707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.521676 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:00.531421 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.531270 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2hh\" (UniqueName: \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.531976 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.531950 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvblt\" (UniqueName: \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.532231 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.532204 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:00.532818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.532796 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwxt\" (UniqueName: \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.533453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.533425 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whd6p\" (UniqueName: \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\") pod \"maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.533618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.533597 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsh6\" (UniqueName: \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\") pod \"maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.533802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.533767 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczhj\" (UniqueName: \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\") pod \"maas-api-key-cleanup-29895780-w8rfh\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:00.537119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.537094 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbzx\" (UniqueName: \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.574060 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.573251 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:00.585869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.585651 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:00.599290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.599224 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:00.605678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.604374 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:00.618455 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.618401 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:00.649669 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.649363 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:00.654104 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:00.653911 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:01.130847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.130819 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:01.131306 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.131284 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27857e16_2d24_4253_b3e0_6e02e1d6442d.slice/crio-ff792ea718d806fd162e87358965b32333465d6bc1fea590bc7611eec98c8c14 WatchSource:0}: Error finding container ff792ea718d806fd162e87358965b32333465d6bc1fea590bc7611eec98c8c14: Status 404 returned error can't find the container with id ff792ea718d806fd162e87358965b32333465d6bc1fea590bc7611eec98c8c14 Nov 03 23:00:01.183052 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.183004 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f3a7a5_23e0_4c0b_b3cf_ec6165387fcc.slice/crio-ad7878c0aff8bbe77543ae6eb000b8d3748dd11d5c093ec44b5c45b70833acd4 WatchSource:0}: Error finding container ad7878c0aff8bbe77543ae6eb000b8d3748dd11d5c093ec44b5c45b70833acd4: Status 404 returned error can't find the container with id ad7878c0aff8bbe77543ae6eb000b8d3748dd11d5c093ec44b5c45b70833acd4 Nov 03 23:00:01.183419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.183334 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:01.184448 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.184423 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b524507_ace4_41fc_b16d_f97b04bd220f.slice/crio-49c4c56510e624fd905c3fc68ff3b326ed8818a4b613dbcc1d8eea2c3b4d7a3d WatchSource:0}: Error finding container 49c4c56510e624fd905c3fc68ff3b326ed8818a4b613dbcc1d8eea2c3b4d7a3d: Status 404 returned error can't find the container with id 49c4c56510e624fd905c3fc68ff3b326ed8818a4b613dbcc1d8eea2c3b4d7a3d Nov 03 23:00:01.188257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.188234 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:01.214958 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.214932 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerStarted","Data":"ff792ea718d806fd162e87358965b32333465d6bc1fea590bc7611eec98c8c14"} Nov 03 23:00:01.216547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.216518 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerStarted","Data":"ad7878c0aff8bbe77543ae6eb000b8d3748dd11d5c093ec44b5c45b70833acd4"} Nov 03 23:00:01.218025 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.217982 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerStarted","Data":"49c4c56510e624fd905c3fc68ff3b326ed8818a4b613dbcc1d8eea2c3b4d7a3d"} Nov 03 23:00:01.256808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.256577 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:01.285343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.285319 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:01.295798 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.295773 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:01.309827 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.309802 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:01.347054 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.347024 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:01.370273 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.370252 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:01.374762 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.374500 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21513ba_a54a_4e82_afcc_8256f0ce92a8.slice/crio-f0f7b52d30228309c757a8654268ccac1e3164103a00bff0e6cb4120d9275b94 WatchSource:0}: Error finding container f0f7b52d30228309c757a8654268ccac1e3164103a00bff0e6cb4120d9275b94: Status 404 returned error can't find the container with id f0f7b52d30228309c757a8654268ccac1e3164103a00bff0e6cb4120d9275b94 Nov 03 23:00:01.374762 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.374701 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d208616_230b_40b2_8a40_56887671a1d0.slice/crio-8f325fcdc5a0e3e0fa75fc10f19943a7adcbc0af86c1585c9584b76d37883b5d WatchSource:0}: Error finding container 8f325fcdc5a0e3e0fa75fc10f19943a7adcbc0af86c1585c9584b76d37883b5d: Status 404 returned error can't find the container with id 8f325fcdc5a0e3e0fa75fc10f19943a7adcbc0af86c1585c9584b76d37883b5d Nov 03 23:00:01.375245 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.375208 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244c650a_006c_424d_9378_324f831189bf.slice/crio-c8c54d52a75e38ecf44c2c481d8ec1e1529868721d4a4a55271bb19daac9a524 WatchSource:0}: Error finding container c8c54d52a75e38ecf44c2c481d8ec1e1529868721d4a4a55271bb19daac9a524: Status 404 returned error can't find the container with id c8c54d52a75e38ecf44c2c481d8ec1e1529868721d4a4a55271bb19daac9a524 Nov 03 23:00:01.428118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:01.428091 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:01.438582 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:01.438554 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadfa6d5_cd89_4810_ae1a_928eb3439204.slice/crio-2b6e8d9e32af3214596e656fb1c20be7ec1407589cf0c332362cdae14adb1419 WatchSource:0}: Error finding container 2b6e8d9e32af3214596e656fb1c20be7ec1407589cf0c332362cdae14adb1419: Status 404 returned error can't find the container with id 2b6e8d9e32af3214596e656fb1c20be7ec1407589cf0c332362cdae14adb1419 Nov 03 23:00:02.232152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.232049 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerStarted","Data":"c8c54d52a75e38ecf44c2c481d8ec1e1529868721d4a4a55271bb19daac9a524"} Nov 03 23:00:02.234199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.234168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerStarted","Data":"fa52d8dd432c6e3802386aee9f1006540b8524facd3491d650929ab1cc7ddacd"} Nov 03 23:00:02.234405 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.234223 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerStarted","Data":"a9ae56a15199c54c91b3812d03fe51168b51d900d4727ac6c90de2dc965bafae"} Nov 03 23:00:02.235942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.235916 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerStarted","Data":"fd680535553992bd7c6562c0f880e13a518b99e9dedc95a6e66c52aba7dd18a1"} Nov 03 23:00:02.237218 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.237195 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerStarted","Data":"8f325fcdc5a0e3e0fa75fc10f19943a7adcbc0af86c1585c9584b76d37883b5d"} Nov 03 23:00:02.238794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.238772 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerStarted","Data":"0a1bb7e81f82a46a9f2fbfc915380114c54cd0f881e7086593c32d39460c2cd6"} Nov 03 23:00:02.240297 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.240275 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerStarted","Data":"0799e1bad7eae65351337ad561c33d74d2f52bed0cc3434544a56cb9547a64b7"} Nov 03 23:00:02.241717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.241698 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerStarted","Data":"91b98f5177dea604deb242518cf5f73eb6106aa47fa16ec1ac64787370cf23a3"} Nov 03 23:00:02.241794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.241738 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerStarted","Data":"1265987ce2b98849b19ff3f408f37f7d75ca14a7d11595b2e07aa5bfc4bafc2c"} Nov 03 23:00:02.243241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.243217 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerStarted","Data":"d90a23dc4c511f68f6a28ea5b63c0620e35973ca8e08c0fc1c3a04753d8ba5db"} Nov 03 23:00:02.243342 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.243266 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerStarted","Data":"2b6e8d9e32af3214596e656fb1c20be7ec1407589cf0c332362cdae14adb1419"} Nov 03 23:00:02.244523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.244503 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerStarted","Data":"f0f7b52d30228309c757a8654268ccac1e3164103a00bff0e6cb4120d9275b94"} Nov 03 23:00:02.246183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.246164 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerStarted","Data":"6a12213646feaadf403634163eed446a01112ede9a456a702d8372507176f7fb"} Nov 03 23:00:02.246270 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.246197 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerStarted","Data":"2d4dce74bd5666835168257cc8bb51dd25e9dbae572d866c8b49e8af849142a4"} Nov 03 23:00:02.350742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.350683 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" podStartSLOduration=2.350665747 podStartE2EDuration="2.350665747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.347943251 +0000 UTC m=+2276.085287476" watchObservedRunningTime="2026-11-03 23:00:02.350665747 +0000 UTC m=+2276.088009972" Nov 03 23:00:02.475740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.475688 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" podStartSLOduration=2.475672987 podStartE2EDuration="2.475672987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.471053667 +0000 UTC m=+2276.208397906" watchObservedRunningTime="2026-11-03 23:00:02.475672987 +0000 UTC m=+2276.213017211" Nov 03 23:00:02.588889 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.588798 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" podStartSLOduration=2.5887801489999998 podStartE2EDuration="2.588780149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.586197736 +0000 UTC m=+2276.323541962" watchObservedRunningTime="2026-11-03 23:00:02.588780149 +0000 UTC m=+2276.326124375" Nov 03 23:00:02.705733 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.705678 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" podStartSLOduration=2.7056646 podStartE2EDuration="2.7056646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.700437112 +0000 UTC m=+2276.437781335" watchObservedRunningTime="2026-11-03 23:00:02.7056646 +0000 UTC m=+2276.443008824" Nov 03 23:00:02.820880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.820785 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" podStartSLOduration=2.820768499 podStartE2EDuration="2.820768499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.81863441 +0000 UTC m=+2276.555978635" watchObservedRunningTime="2026-11-03 23:00:02.820768499 +0000 UTC m=+2276.558112724" Nov 03 23:00:02.983660 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:02.983598 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" podStartSLOduration=2.9835797360000003 podStartE2EDuration="2.983579736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.979259902 +0000 UTC m=+2276.716604126" watchObservedRunningTime="2026-11-03 23:00:02.983579736 +0000 UTC m=+2276.720923962" Nov 03 23:00:03.097452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.097322 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" podStartSLOduration=3.097303034 podStartE2EDuration="3.097303034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.093046912 +0000 UTC m=+2276.830391136" watchObservedRunningTime="2026-11-03 23:00:03.097303034 +0000 UTC m=+2276.834647261" Nov 03 23:00:03.257395 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.257359 5051 generic.go:356] "Generic (PLEG): container finished" podID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerID="0a1bb7e81f82a46a9f2fbfc915380114c54cd0f881e7086593c32d39460c2cd6" exitCode=60 Nov 03 23:00:03.257914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.257481 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerDied","Data":"0a1bb7e81f82a46a9f2fbfc915380114c54cd0f881e7086593c32d39460c2cd6"} Nov 03 23:00:03.257914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.257827 5051 scope.go:122] "RemoveContainer" containerID="0a1bb7e81f82a46a9f2fbfc915380114c54cd0f881e7086593c32d39460c2cd6" Nov 03 23:00:03.259891 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.259870 5051 generic.go:356] "Generic (PLEG): container finished" podID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerID="0799e1bad7eae65351337ad561c33d74d2f52bed0cc3434544a56cb9547a64b7" exitCode=60 Nov 03 23:00:03.260041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.259983 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerDied","Data":"0799e1bad7eae65351337ad561c33d74d2f52bed0cc3434544a56cb9547a64b7"} Nov 03 23:00:03.260420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.260405 5051 scope.go:122] "RemoveContainer" containerID="0799e1bad7eae65351337ad561c33d74d2f52bed0cc3434544a56cb9547a64b7" Nov 03 23:00:03.273736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.273260 5051 generic.go:356] "Generic (PLEG): container finished" podID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerID="91b98f5177dea604deb242518cf5f73eb6106aa47fa16ec1ac64787370cf23a3" exitCode=60 Nov 03 23:00:03.273736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.273350 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerDied","Data":"91b98f5177dea604deb242518cf5f73eb6106aa47fa16ec1ac64787370cf23a3"} Nov 03 23:00:03.273736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.273674 5051 scope.go:122] "RemoveContainer" containerID="91b98f5177dea604deb242518cf5f73eb6106aa47fa16ec1ac64787370cf23a3" Nov 03 23:00:03.275684 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.275463 5051 generic.go:356] "Generic (PLEG): container finished" podID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerID="d90a23dc4c511f68f6a28ea5b63c0620e35973ca8e08c0fc1c3a04753d8ba5db" exitCode=60 Nov 03 23:00:03.275684 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.275554 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerDied","Data":"d90a23dc4c511f68f6a28ea5b63c0620e35973ca8e08c0fc1c3a04753d8ba5db"} Nov 03 23:00:03.275811 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.275757 5051 scope.go:122] "RemoveContainer" containerID="d90a23dc4c511f68f6a28ea5b63c0620e35973ca8e08c0fc1c3a04753d8ba5db" Nov 03 23:00:03.278261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.278239 5051 generic.go:356] "Generic (PLEG): container finished" podID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerID="2e3d58cd15533ec83bbf9f8c27ac76207d19c35640be26d00dc041402e1f6494" exitCode=60 Nov 03 23:00:03.278433 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.278274 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerDied","Data":"2e3d58cd15533ec83bbf9f8c27ac76207d19c35640be26d00dc041402e1f6494"} Nov 03 23:00:03.278559 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.278542 5051 scope.go:122] "RemoveContainer" containerID="2e3d58cd15533ec83bbf9f8c27ac76207d19c35640be26d00dc041402e1f6494" Nov 03 23:00:03.280759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.280736 5051 generic.go:356] "Generic (PLEG): container finished" podID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerID="6a12213646feaadf403634163eed446a01112ede9a456a702d8372507176f7fb" exitCode=60 Nov 03 23:00:03.280882 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.280800 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerDied","Data":"6a12213646feaadf403634163eed446a01112ede9a456a702d8372507176f7fb"} Nov 03 23:00:03.281062 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.281041 5051 scope.go:122] "RemoveContainer" containerID="6a12213646feaadf403634163eed446a01112ede9a456a702d8372507176f7fb" Nov 03 23:00:03.423069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.417876 5051 generic.go:356] "Generic (PLEG): container finished" podID="244c650a-006c-424d-9378-324f831189bf" containerID="3aa023272bb85757beefaea60a994c4852dfbc2b3f011ab310c8fbe0714c0fb7" exitCode=60 Nov 03 23:00:03.423069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.418096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerDied","Data":"3aa023272bb85757beefaea60a994c4852dfbc2b3f011ab310c8fbe0714c0fb7"} Nov 03 23:00:03.423069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.418661 5051 scope.go:122] "RemoveContainer" containerID="3aa023272bb85757beefaea60a994c4852dfbc2b3f011ab310c8fbe0714c0fb7" Nov 03 23:00:03.453435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.453288 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4691626-a281-4473-bf97-7bb4684992dc" containerID="fa52d8dd432c6e3802386aee9f1006540b8524facd3491d650929ab1cc7ddacd" exitCode=60 Nov 03 23:00:03.453782 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.453757 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerDied","Data":"fa52d8dd432c6e3802386aee9f1006540b8524facd3491d650929ab1cc7ddacd"} Nov 03 23:00:03.454675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.454540 5051 scope.go:122] "RemoveContainer" containerID="fa52d8dd432c6e3802386aee9f1006540b8524facd3491d650929ab1cc7ddacd" Nov 03 23:00:03.461781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.461752 5051 generic.go:356] "Generic (PLEG): container finished" podID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerID="fd680535553992bd7c6562c0f880e13a518b99e9dedc95a6e66c52aba7dd18a1" exitCode=60 Nov 03 23:00:03.462649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.461923 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerDied","Data":"fd680535553992bd7c6562c0f880e13a518b99e9dedc95a6e66c52aba7dd18a1"} Nov 03 23:00:03.464419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.462430 5051 scope.go:122] "RemoveContainer" containerID="fd680535553992bd7c6562c0f880e13a518b99e9dedc95a6e66c52aba7dd18a1" Nov 03 23:00:03.468245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.468215 5051 generic.go:356] "Generic (PLEG): container finished" podID="2d208616-230b-40b2-8a40-56887671a1d0" containerID="7b514f9c0b17b85f2fc0b082fbc59a31e00d51df4bde0d99127cd25e5c4e6b03" exitCode=60 Nov 03 23:00:03.468345 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.468265 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerDied","Data":"7b514f9c0b17b85f2fc0b082fbc59a31e00d51df4bde0d99127cd25e5c4e6b03"} Nov 03 23:00:03.469260 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:03.469244 5051 scope.go:122] "RemoveContainer" containerID="7b514f9c0b17b85f2fc0b082fbc59a31e00d51df4bde0d99127cd25e5c4e6b03" Nov 03 23:00:04.480953 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.480920 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerStarted","Data":"39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364"} Nov 03 23:00:04.483441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.483404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerStarted","Data":"62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea"} Nov 03 23:00:04.485766 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.485735 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerStarted","Data":"c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72"} Nov 03 23:00:04.488159 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.488137 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerStarted","Data":"89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323"} Nov 03 23:00:04.490696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.490668 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerStarted","Data":"e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd"} Nov 03 23:00:04.493367 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.493344 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerStarted","Data":"1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122"} Nov 03 23:00:04.642111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:04.642011 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" podStartSLOduration=4.641984687 podStartE2EDuration="4.641984687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.639338243 +0000 UTC m=+2278.376682468" watchObservedRunningTime="2026-11-03 23:00:04.641984687 +0000 UTC m=+2278.379328933" Nov 03 23:00:05.518775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.518687 5051 generic.go:356] "Generic (PLEG): container finished" podID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerID="62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea" exitCode=60 Nov 03 23:00:05.518775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.518744 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerDied","Data":"62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea"} Nov 03 23:00:05.519317 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.518833 5051 scope.go:122] "RemoveContainer" containerID="2e3d58cd15533ec83bbf9f8c27ac76207d19c35640be26d00dc041402e1f6494" Nov 03 23:00:05.519317 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.519229 5051 scope.go:122] "RemoveContainer" containerID="62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea" Nov 03 23:00:05.519636 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.519607 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k_odh-ai-gateway-infra(e21513ba-a54a-4e82-afcc-8256f0ce92a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" Nov 03 23:00:05.521554 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.521536 5051 generic.go:356] "Generic (PLEG): container finished" podID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerID="c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72" exitCode=60 Nov 03 23:00:05.521664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.521629 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerDied","Data":"c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72"} Nov 03 23:00:05.521875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.521858 5051 scope.go:122] "RemoveContainer" containerID="c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72" Nov 03 23:00:05.522160 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.522134 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-w8rfh_odh-ai-gateway-infra(ace9b8d8-a57b-4125-9b80-cb09625afe80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" Nov 03 23:00:05.524176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.524157 5051 generic.go:356] "Generic (PLEG): container finished" podID="244c650a-006c-424d-9378-324f831189bf" containerID="c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914" exitCode=60 Nov 03 23:00:05.524269 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.524184 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerDied","Data":"c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914"} Nov 03 23:00:05.524549 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.524531 5051 scope.go:122] "RemoveContainer" containerID="c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914" Nov 03 23:00:05.524813 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.524792 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs_odh-ai-gateway-infra(244c650a-006c-424d-9378-324f831189bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" podUID="244c650a-006c-424d-9378-324f831189bf" Nov 03 23:00:05.526847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.526825 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4691626-a281-4473-bf97-7bb4684992dc" containerID="595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541" exitCode=60 Nov 03 23:00:05.526956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.526917 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerDied","Data":"595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541"} Nov 03 23:00:05.527177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.527157 5051 scope.go:122] "RemoveContainer" containerID="595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541" Nov 03 23:00:05.527469 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.527433 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq_odh-ai-gateway-infra(d4691626-a281-4473-bf97-7bb4684992dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" podUID="d4691626-a281-4473-bf97-7bb4684992dc" Nov 03 23:00:05.530522 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.530499 5051 generic.go:356] "Generic (PLEG): container finished" podID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerID="1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f" exitCode=60 Nov 03 23:00:05.530614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.530586 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerDied","Data":"1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f"} Nov 03 23:00:05.530789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.530773 5051 scope.go:122] "RemoveContainer" containerID="1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f" Nov 03 23:00:05.531052 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.531022 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd_odh-ai-gateway-infra(27857e16-2d24-4253-b3e0-6e02e1d6442d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" Nov 03 23:00:05.532647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.532628 5051 generic.go:356] "Generic (PLEG): container finished" podID="2d208616-230b-40b2-8a40-56887671a1d0" containerID="068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8" exitCode=60 Nov 03 23:00:05.532749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.532703 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerDied","Data":"068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8"} Nov 03 23:00:05.532886 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.532869 5051 scope.go:122] "RemoveContainer" containerID="068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8" Nov 03 23:00:05.533129 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.533108 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2_odh-ai-gateway-infra(2d208616-230b-40b2-8a40-56887671a1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" podUID="2d208616-230b-40b2-8a40-56887671a1d0" Nov 03 23:00:05.534798 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.534777 5051 generic.go:356] "Generic (PLEG): container finished" podID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerID="89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323" exitCode=60 Nov 03 23:00:05.534876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.534858 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerDied","Data":"89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323"} Nov 03 23:00:05.535046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.535027 5051 scope.go:122] "RemoveContainer" containerID="89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323" Nov 03 23:00:05.535255 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.535235 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4_odh-ai-gateway-infra(40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" Nov 03 23:00:05.536487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.536472 5051 scope.go:122] "RemoveContainer" containerID="6a12213646feaadf403634163eed446a01112ede9a456a702d8372507176f7fb" Nov 03 23:00:05.536784 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.536766 5051 generic.go:356] "Generic (PLEG): container finished" podID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerID="e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd" exitCode=60 Nov 03 23:00:05.536866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.536843 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerDied","Data":"e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd"} Nov 03 23:00:05.537100 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.537082 5051 scope.go:122] "RemoveContainer" containerID="e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd" Nov 03 23:00:05.537324 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.537305 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74_odh-ai-gateway-infra(4b524507-ace4-41fc-b16d-f97b04bd220f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" Nov 03 23:00:05.538849 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.538830 5051 generic.go:356] "Generic (PLEG): container finished" podID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerID="1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122" exitCode=60 Nov 03 23:00:05.538957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.538903 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerDied","Data":"1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122"} Nov 03 23:00:05.539122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.539105 5051 scope.go:122] "RemoveContainer" containerID="1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122" Nov 03 23:00:05.539298 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.539281 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7_odh-ai-gateway-infra(4ec95a0c-34f7-46fc-b52e-86267cf6570f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" Nov 03 23:00:05.540642 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.540624 5051 generic.go:356] "Generic (PLEG): container finished" podID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerID="39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364" exitCode=60 Nov 03 23:00:05.540711 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.540663 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerDied","Data":"39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364"} Nov 03 23:00:05.540859 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.540845 5051 scope.go:122] "RemoveContainer" containerID="39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364" Nov 03 23:00:05.541060 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:05.541044 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg_odh-ai-gateway-infra(dadfa6d5-cd89-4810-ae1a-928eb3439204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" Nov 03 23:00:05.550490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.550472 5051 scope.go:122] "RemoveContainer" containerID="3aa023272bb85757beefaea60a994c4852dfbc2b3f011ab310c8fbe0714c0fb7" Nov 03 23:00:05.564183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.564162 5051 scope.go:122] "RemoveContainer" containerID="fa52d8dd432c6e3802386aee9f1006540b8524facd3491d650929ab1cc7ddacd" Nov 03 23:00:05.579809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.579787 5051 scope.go:122] "RemoveContainer" containerID="fd680535553992bd7c6562c0f880e13a518b99e9dedc95a6e66c52aba7dd18a1" Nov 03 23:00:05.596681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.596644 5051 scope.go:122] "RemoveContainer" containerID="7b514f9c0b17b85f2fc0b082fbc59a31e00d51df4bde0d99127cd25e5c4e6b03" Nov 03 23:00:05.612789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.612770 5051 scope.go:122] "RemoveContainer" containerID="0a1bb7e81f82a46a9f2fbfc915380114c54cd0f881e7086593c32d39460c2cd6" Nov 03 23:00:05.627654 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.627631 5051 scope.go:122] "RemoveContainer" containerID="0799e1bad7eae65351337ad561c33d74d2f52bed0cc3434544a56cb9547a64b7" Nov 03 23:00:05.643323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.643304 5051 scope.go:122] "RemoveContainer" containerID="91b98f5177dea604deb242518cf5f73eb6106aa47fa16ec1ac64787370cf23a3" Nov 03 23:00:05.658323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:05.658301 5051 scope.go:122] "RemoveContainer" containerID="d90a23dc4c511f68f6a28ea5b63c0620e35973ca8e08c0fc1c3a04753d8ba5db" Nov 03 23:00:06.551547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.551517 5051 scope.go:122] "RemoveContainer" containerID="1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f" Nov 03 23:00:06.552008 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.551886 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd_odh-ai-gateway-infra(27857e16-2d24-4253-b3e0-6e02e1d6442d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" Nov 03 23:00:06.553096 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.553079 5051 scope.go:122] "RemoveContainer" containerID="068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8" Nov 03 23:00:06.553366 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.553341 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2_odh-ai-gateway-infra(2d208616-230b-40b2-8a40-56887671a1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" podUID="2d208616-230b-40b2-8a40-56887671a1d0" Nov 03 23:00:06.554743 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.554724 5051 scope.go:122] "RemoveContainer" containerID="89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323" Nov 03 23:00:06.555027 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.554971 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4_odh-ai-gateway-infra(40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" Nov 03 23:00:06.556228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.556199 5051 scope.go:122] "RemoveContainer" containerID="e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd" Nov 03 23:00:06.556472 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.556451 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74_odh-ai-gateway-infra(4b524507-ace4-41fc-b16d-f97b04bd220f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" Nov 03 23:00:06.557815 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.557795 5051 scope.go:122] "RemoveContainer" containerID="1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122" Nov 03 23:00:06.558091 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.558067 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7_odh-ai-gateway-infra(4ec95a0c-34f7-46fc-b52e-86267cf6570f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" Nov 03 23:00:06.559428 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.559410 5051 scope.go:122] "RemoveContainer" containerID="39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364" Nov 03 23:00:06.559648 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.559627 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg_odh-ai-gateway-infra(dadfa6d5-cd89-4810-ae1a-928eb3439204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" Nov 03 23:00:06.561106 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.561086 5051 scope.go:122] "RemoveContainer" containerID="62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea" Nov 03 23:00:06.561329 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.561308 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k_odh-ai-gateway-infra(e21513ba-a54a-4e82-afcc-8256f0ce92a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" Nov 03 23:00:06.562818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.562799 5051 scope.go:122] "RemoveContainer" containerID="c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72" Nov 03 23:00:06.563093 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.563061 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-w8rfh_odh-ai-gateway-infra(ace9b8d8-a57b-4125-9b80-cb09625afe80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" Nov 03 23:00:06.564585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.564565 5051 scope.go:122] "RemoveContainer" containerID="c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914" Nov 03 23:00:06.564830 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.564799 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs_odh-ai-gateway-infra(244c650a-006c-424d-9378-324f831189bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" podUID="244c650a-006c-424d-9378-324f831189bf" Nov 03 23:00:06.566245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:06.566230 5051 scope.go:122] "RemoveContainer" containerID="595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541" Nov 03 23:00:06.566429 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:06.566412 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq_odh-ai-gateway-infra(d4691626-a281-4473-bf97-7bb4684992dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" podUID="d4691626-a281-4473-bf97-7bb4684992dc" Nov 03 23:00:15.153198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.153072 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:15.454288 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.454261 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:15.498597 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.498561 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\" (UniqueName: \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\") pod \"4b524507-ace4-41fc-b16d-f97b04bd220f\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " Nov 03 23:00:15.498807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.498624 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\") pod \"4b524507-ace4-41fc-b16d-f97b04bd220f\" (UID: \"4b524507-ace4-41fc-b16d-f97b04bd220f\") " Nov 03 23:00:15.499053 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.499024 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca" pod "4b524507-ace4-41fc-b16d-f97b04bd220f" (UID: "4b524507-ace4-41fc-b16d-f97b04bd220f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:15.499529 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.499454 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b524507-ace4-41fc-b16d-f97b04bd220f-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.501066 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.501044 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt" pod "4b524507-ace4-41fc-b16d-f97b04bd220f" (UID: "4b524507-ace4-41fc-b16d-f97b04bd220f"). InnerVolumeSpecName "kube-api-access-9w9wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:15.607403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.600482 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9wt\" (UniqueName: \"kubernetes.io/projected/4b524507-ace4-41fc-b16d-f97b04bd220f-kube-api-access-9w9wt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.650388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.650351 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" Nov 03 23:00:15.650388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.650374 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74" event={"ID":"4b524507-ace4-41fc-b16d-f97b04bd220f","Type":"ContainerDied","Data":"49c4c56510e624fd905c3fc68ff3b326ed8818a4b613dbcc1d8eea2c3b4d7a3d"} Nov 03 23:00:15.650609 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.650468 5051 scope.go:122] "RemoveContainer" containerID="e791a858e0102fbe14f26511ff90c8303004821272e4f6856ab61107b86bc0dd" Nov 03 23:00:15.817028 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.816957 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:15.823299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:15.823266 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-cbd26c-29895780-mlg74"] Nov 03 23:00:16.051267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.051186 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4"] Nov 03 23:00:16.058177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.058143 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a62b53-jxbv4"] Nov 03 23:00:16.406220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.406194 5051 scope.go:122] "RemoveContainer" containerID="c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72" Nov 03 23:00:16.406729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.406312 5051 scope.go:122] "RemoveContainer" containerID="39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364" Nov 03 23:00:16.406729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.406426 5051 scope.go:122] "RemoveContainer" containerID="89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323" Nov 03 23:00:16.406729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.406498 5051 scope.go:122] "RemoveContainer" containerID="595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541" Nov 03 23:00:16.411360 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.411338 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" path="/var/lib/kubelet/pods/4b524507-ace4-41fc-b16d-f97b04bd220f/volumes" Nov 03 23:00:16.411766 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:16.411753 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3e1282-5769-412e-9055-eeec76d312e9" path="/var/lib/kubelet/pods/8d3e1282-5769-412e-9055-eeec76d312e9/volumes" Nov 03 23:00:17.400696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.400670 5051 scope.go:122] "RemoveContainer" containerID="1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f" Nov 03 23:00:17.400881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.400841 5051 scope.go:122] "RemoveContainer" containerID="c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914" Nov 03 23:00:17.401045 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.401021 5051 scope.go:122] "RemoveContainer" containerID="1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122" Nov 03 23:00:17.680311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.680280 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerStarted","Data":"46e1151a55221cb3f2328fb8c038b1f0d80151d464660f83be79a3903d22f360"} Nov 03 23:00:17.683474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.683446 5051 generic.go:356] "Generic (PLEG): container finished" podID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerID="203d95642aae0135b325df6b12878c0256490a10daf8ec10e5828afcc578cac5" exitCode=60 Nov 03 23:00:17.683612 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.683513 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerDied","Data":"203d95642aae0135b325df6b12878c0256490a10daf8ec10e5828afcc578cac5"} Nov 03 23:00:17.683612 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.683585 5051 scope.go:122] "RemoveContainer" containerID="c71c03a7b39fa7f7a75595984519f6e908e03ac81788b6ad021465d5b0a2ee72" Nov 03 23:00:17.683973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.683956 5051 scope.go:122] "RemoveContainer" containerID="203d95642aae0135b325df6b12878c0256490a10daf8ec10e5828afcc578cac5" Nov 03 23:00:17.684341 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:17.684322 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895780-w8rfh_odh-ai-gateway-infra(ace9b8d8-a57b-4125-9b80-cb09625afe80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" Nov 03 23:00:17.686580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.686561 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerStarted","Data":"b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254"} Nov 03 23:00:17.686872 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.686858 5051 scope.go:122] "RemoveContainer" containerID="b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254" Nov 03 23:00:17.687174 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:17.687155 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq_odh-ai-gateway-infra(d4691626-a281-4473-bf97-7bb4684992dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" podUID="d4691626-a281-4473-bf97-7bb4684992dc" Nov 03 23:00:17.689756 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.689733 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerStarted","Data":"764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5"} Nov 03 23:00:17.690142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.690110 5051 scope.go:122] "RemoveContainer" containerID="764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5" Nov 03 23:00:17.690455 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:17.690432 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4_odh-ai-gateway-infra(40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" Nov 03 23:00:17.718982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.718948 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf"] Nov 03 23:00:17.723287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:17.723259 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-cbd26c-8bmwf"] Nov 03 23:00:18.416631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.416586 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e" path="/var/lib/kubelet/pods/2703b0a9-4a81-4924-9f1d-6ea9a7d07c7e/volumes" Nov 03 23:00:18.418169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.418139 5051 scope.go:122] "RemoveContainer" containerID="068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8" Nov 03 23:00:18.419501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.419483 5051 scope.go:122] "RemoveContainer" containerID="62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea" Nov 03 23:00:18.703761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.703639 5051 generic.go:356] "Generic (PLEG): container finished" podID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerID="764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5" exitCode=60 Nov 03 23:00:18.703761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.703694 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerDied","Data":"764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5"} Nov 03 23:00:18.703761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.703783 5051 scope.go:122] "RemoveContainer" containerID="89dafe4bb6eecb47018de2bd21b6a08ac608082ec1bea24f6a7a17ef8d2f4323" Nov 03 23:00:18.704418 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.704301 5051 scope.go:122] "RemoveContainer" containerID="764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5" Nov 03 23:00:18.704815 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:18.704783 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4_odh-ai-gateway-infra(40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" Nov 03 23:00:18.707327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.707304 5051 generic.go:356] "Generic (PLEG): container finished" podID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerID="698de4832b23c18e62d464992a104a7051cdcc2b89b26f956b788f0f6b9f659d" exitCode=60 Nov 03 23:00:18.707445 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.707380 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerDied","Data":"698de4832b23c18e62d464992a104a7051cdcc2b89b26f956b788f0f6b9f659d"} Nov 03 23:00:18.707799 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.707781 5051 scope.go:122] "RemoveContainer" containerID="698de4832b23c18e62d464992a104a7051cdcc2b89b26f956b788f0f6b9f659d" Nov 03 23:00:18.708240 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:18.708203 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7_odh-ai-gateway-infra(4ec95a0c-34f7-46fc-b52e-86267cf6570f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" Nov 03 23:00:18.709513 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.709490 5051 generic.go:356] "Generic (PLEG): container finished" podID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerID="46e1151a55221cb3f2328fb8c038b1f0d80151d464660f83be79a3903d22f360" exitCode=60 Nov 03 23:00:18.709614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.709555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerDied","Data":"46e1151a55221cb3f2328fb8c038b1f0d80151d464660f83be79a3903d22f360"} Nov 03 23:00:18.709972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.709949 5051 scope.go:122] "RemoveContainer" containerID="46e1151a55221cb3f2328fb8c038b1f0d80151d464660f83be79a3903d22f360" Nov 03 23:00:18.710397 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:18.710376 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg_odh-ai-gateway-infra(dadfa6d5-cd89-4810-ae1a-928eb3439204)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" Nov 03 23:00:18.713299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.713280 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerStarted","Data":"56ffe3d9d5a96d246f65b47c11a6242f52e259dcfab9c4e17b73b67936e35d71"} Nov 03 23:00:18.715847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.715827 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4691626-a281-4473-bf97-7bb4684992dc" containerID="b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254" exitCode=60 Nov 03 23:00:18.715932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.715905 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerDied","Data":"b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254"} Nov 03 23:00:18.716254 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.716231 5051 scope.go:122] "RemoveContainer" containerID="b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254" Nov 03 23:00:18.716574 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:18.716532 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq_odh-ai-gateway-infra(d4691626-a281-4473-bf97-7bb4684992dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" podUID="d4691626-a281-4473-bf97-7bb4684992dc" Nov 03 23:00:18.718050 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.718028 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerStarted","Data":"051dc1a6b7972ee4cb253eab274d8749a09f19362d4a27c73dde18278b2eadb0"} Nov 03 23:00:18.723787 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.723768 5051 scope.go:122] "RemoveContainer" containerID="1e289b5fabbce20141746c29b76bc2cb4b980891a9675e1324d3fa5a3e5f7122" Nov 03 23:00:18.743721 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.743702 5051 scope.go:122] "RemoveContainer" containerID="39944c2b86202b15f48912697bc5ab302a357eb552deb66c569393b8d5d51364" Nov 03 23:00:18.762365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.762341 5051 scope.go:122] "RemoveContainer" containerID="595a32c1fd9d67ae0945d58fdfbc45943bf44d2ebe4e35fc7dbc7447cfd46541" Nov 03 23:00:18.838344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.838312 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:18.953907 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:18.953822 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:19.068848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.068821 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:19.167482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.167426 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" podStartSLOduration=19.167408475 podStartE2EDuration="19.167408475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:19.163876255 +0000 UTC m=+2292.901220480" watchObservedRunningTime="2026-11-03 23:00:19.167408475 +0000 UTC m=+2292.904752700" Nov 03 23:00:19.182524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.182483 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:19.389474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.389440 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:19.460687 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.460653 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\") pod \"ace9b8d8-a57b-4125-9b80-cb09625afe80\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " Nov 03 23:00:19.460878 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.460704 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\" (UniqueName: \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\") pod \"ace9b8d8-a57b-4125-9b80-cb09625afe80\" (UID: \"ace9b8d8-a57b-4125-9b80-cb09625afe80\") " Nov 03 23:00:19.461369 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.461043 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca" pod "ace9b8d8-a57b-4125-9b80-cb09625afe80" (UID: "ace9b8d8-a57b-4125-9b80-cb09625afe80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:19.461511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.461415 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace9b8d8-a57b-4125-9b80-cb09625afe80-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:19.463842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.463816 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj" pod "ace9b8d8-a57b-4125-9b80-cb09625afe80" (UID: "ace9b8d8-a57b-4125-9b80-cb09625afe80"). InnerVolumeSpecName "kube-api-access-bczhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:19.563107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.562975 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczhj\" (UniqueName: \"kubernetes.io/projected/ace9b8d8-a57b-4125-9b80-cb09625afe80-kube-api-access-bczhj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:19.731172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.731141 5051 generic.go:356] "Generic (PLEG): container finished" podID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerID="051dc1a6b7972ee4cb253eab274d8749a09f19362d4a27c73dde18278b2eadb0" exitCode=60 Nov 03 23:00:19.731662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.731213 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerDied","Data":"051dc1a6b7972ee4cb253eab274d8749a09f19362d4a27c73dde18278b2eadb0"} Nov 03 23:00:19.731662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.731276 5051 scope.go:122] "RemoveContainer" containerID="1d48f63c9916cbf22a3033d0f451a4e499460edfa682c902827202d71a92c29f" Nov 03 23:00:19.731820 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.731729 5051 scope.go:122] "RemoveContainer" containerID="051dc1a6b7972ee4cb253eab274d8749a09f19362d4a27c73dde18278b2eadb0" Nov 03 23:00:19.732230 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:19.732202 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd_odh-ai-gateway-infra(27857e16-2d24-4253-b3e0-6e02e1d6442d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" Nov 03 23:00:19.733615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.733593 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerStarted","Data":"8be4bcff0489057d0e48eeddfb2c97d4f55e8b9353cb412d1820abcd740d2851"} Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.764657 5051 generic.go:356] "Generic (PLEG): container finished" podID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerID="25b3f852cbff963d9d0c26d952023fae022248398b20d1d24513bd3bd627a7f6" exitCode=60 Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.764804 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerDied","Data":"25b3f852cbff963d9d0c26d952023fae022248398b20d1d24513bd3bd627a7f6"} Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.769494 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" event={"ID":"ace9b8d8-a57b-4125-9b80-cb09625afe80","Type":"ContainerDied","Data":"2d4dce74bd5666835168257cc8bb51dd25e9dbae572d866c8b49e8af849142a4"} Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.769606 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh" Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.774102 5051 scope.go:122] "RemoveContainer" containerID="25b3f852cbff963d9d0c26d952023fae022248398b20d1d24513bd3bd627a7f6" Nov 03 23:00:19.775763 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:19.774540 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k_odh-ai-gateway-infra(e21513ba-a54a-4e82-afcc-8256f0ce92a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" Nov 03 23:00:19.780007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.779952 5051 generic.go:356] "Generic (PLEG): container finished" podID="244c650a-006c-424d-9378-324f831189bf" containerID="56ffe3d9d5a96d246f65b47c11a6242f52e259dcfab9c4e17b73b67936e35d71" exitCode=60 Nov 03 23:00:19.780155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.780058 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerDied","Data":"56ffe3d9d5a96d246f65b47c11a6242f52e259dcfab9c4e17b73b67936e35d71"} Nov 03 23:00:19.780501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.780480 5051 scope.go:122] "RemoveContainer" containerID="56ffe3d9d5a96d246f65b47c11a6242f52e259dcfab9c4e17b73b67936e35d71" Nov 03 23:00:19.780853 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:19.780825 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs_odh-ai-gateway-infra(244c650a-006c-424d-9378-324f831189bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" podUID="244c650a-006c-424d-9378-324f831189bf" Nov 03 23:00:19.818035 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.818010 5051 scope.go:122] "RemoveContainer" containerID="62713a28f597f498400af740b743f90fcdbb2ce44ae8a5bad0343182041f85ea" Nov 03 23:00:19.840189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.840157 5051 scope.go:122] "RemoveContainer" containerID="203d95642aae0135b325df6b12878c0256490a10daf8ec10e5828afcc578cac5" Nov 03 23:00:19.954440 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.954417 5051 scope.go:122] "RemoveContainer" containerID="c1212d982c3e235aa986f751d5fb8f39e3b34c0ae22d3c834eef507af2a8a914" Nov 03 23:00:19.991380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.991348 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:20.007101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:19.997329 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" podStartSLOduration=19.997310364 podStartE2EDuration="19.997310364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:19.986454886 +0000 UTC m=+2293.723799112" watchObservedRunningTime="2026-11-03 23:00:19.997310364 +0000 UTC m=+2293.734654590" Nov 03 23:00:20.052775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.050401 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:20.076734 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.076624 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:20.076909 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.076884 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\") pod \"d4691626-a281-4473-bf97-7bb4684992dc\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " Nov 03 23:00:20.077224 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.077197 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\" (UniqueName: \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\") pod \"d4691626-a281-4473-bf97-7bb4684992dc\" (UID: \"d4691626-a281-4473-bf97-7bb4684992dc\") " Nov 03 23:00:20.077806 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.077540 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca" pod "d4691626-a281-4473-bf97-7bb4684992dc" (UID: "d4691626-a281-4473-bf97-7bb4684992dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.077944 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.077923 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4691626-a281-4473-bf97-7bb4684992dc-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.080474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.080446 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh" pod "d4691626-a281-4473-bf97-7bb4684992dc" (UID: "d4691626-a281-4473-bf97-7bb4684992dc"). InnerVolumeSpecName "kube-api-access-kz2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.096514 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.096252 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:20.109878 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.109847 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:20.116350 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.116324 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895780-w8rfh"] Nov 03 23:00:20.176803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.176765 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:20.179015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.178968 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\") pod \"dadfa6d5-cd89-4810-ae1a-928eb3439204\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " Nov 03 23:00:20.179146 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179047 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\" (UniqueName: \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\") pod \"dadfa6d5-cd89-4810-ae1a-928eb3439204\" (UID: \"dadfa6d5-cd89-4810-ae1a-928eb3439204\") " Nov 03 23:00:20.179217 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179194 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\") pod \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " Nov 03 23:00:20.179282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179266 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\" (UniqueName: \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\") pod \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\" (UID: \"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc\") " Nov 03 23:00:20.179375 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179348 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca" pod "dadfa6d5-cd89-4810-ae1a-928eb3439204" (UID: "dadfa6d5-cd89-4810-ae1a-928eb3439204"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.179585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179565 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca" pod "40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" (UID: "40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.180417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179950 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadfa6d5-cd89-4810-ae1a-928eb3439204-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.180417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.179979 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2hh\" (UniqueName: \"kubernetes.io/projected/d4691626-a281-4473-bf97-7bb4684992dc-kube-api-access-kz2hh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.180417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.180044 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.182052 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.182028 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p" pod "dadfa6d5-cd89-4810-ae1a-928eb3439204" (UID: "dadfa6d5-cd89-4810-ae1a-928eb3439204"). InnerVolumeSpecName "kube-api-access-whd6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.182354 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.182331 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz" pod "40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" (UID: "40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc"). InnerVolumeSpecName "kube-api-access-q9ssz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.281128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.281091 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whd6p\" (UniqueName: \"kubernetes.io/projected/dadfa6d5-cd89-4810-ae1a-928eb3439204-kube-api-access-whd6p\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.281128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.281121 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ssz\" (UniqueName: \"kubernetes.io/projected/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc-kube-api-access-q9ssz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.293745 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.293714 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:20.413507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.413467 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" path="/var/lib/kubelet/pods/ace9b8d8-a57b-4125-9b80-cb09625afe80/volumes" Nov 03 23:00:20.451954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.451931 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:20.483781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.483754 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\" (UniqueName: \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\") pod \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " Nov 03 23:00:20.483932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.483837 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\") pod \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\" (UID: \"4ec95a0c-34f7-46fc-b52e-86267cf6570f\") " Nov 03 23:00:20.484295 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.484268 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca" pod "4ec95a0c-34f7-46fc-b52e-86267cf6570f" (UID: "4ec95a0c-34f7-46fc-b52e-86267cf6570f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.484534 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.484517 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec95a0c-34f7-46fc-b52e-86267cf6570f-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:20.486125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.486104 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx" pod "4ec95a0c-34f7-46fc-b52e-86267cf6570f" (UID: "4ec95a0c-34f7-46fc-b52e-86267cf6570f"). InnerVolumeSpecName "kube-api-access-7fbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.587355 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:20.587313 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbzx\" (UniqueName: \"kubernetes.io/projected/4ec95a0c-34f7-46fc-b52e-86267cf6570f-kube-api-access-7fbzx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.036296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.036259 5051 generic.go:356] "Generic (PLEG): container finished" podID="2d208616-230b-40b2-8a40-56887671a1d0" containerID="8be4bcff0489057d0e48eeddfb2c97d4f55e8b9353cb412d1820abcd740d2851" exitCode=60 Nov 03 23:00:21.036875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.036295 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerDied","Data":"8be4bcff0489057d0e48eeddfb2c97d4f55e8b9353cb412d1820abcd740d2851"} Nov 03 23:00:21.036875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.036815 5051 scope.go:122] "RemoveContainer" containerID="068664ae1a9af0631bc6cad76a1caef731655a2c05f99b326d6ea2cc1866e5d8" Nov 03 23:00:21.037220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.036950 5051 scope.go:122] "RemoveContainer" containerID="8be4bcff0489057d0e48eeddfb2c97d4f55e8b9353cb412d1820abcd740d2851" Nov 03 23:00:21.038664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.038355 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" Nov 03 23:00:21.039070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.039047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4" event={"ID":"40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc","Type":"ContainerDied","Data":"ad7878c0aff8bbe77543ae6eb000b8d3748dd11d5c093ec44b5c45b70833acd4"} Nov 03 23:00:21.040315 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:21.040276 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2_odh-ai-gateway-infra(2d208616-230b-40b2-8a40-56887671a1d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" podUID="2d208616-230b-40b2-8a40-56887671a1d0" Nov 03 23:00:21.042298 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.042271 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:21.045607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.045579 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" event={"ID":"4ec95a0c-34f7-46fc-b52e-86267cf6570f","Type":"ContainerDied","Data":"1265987ce2b98849b19ff3f408f37f7d75ca14a7d11595b2e07aa5bfc4bafc2c"} Nov 03 23:00:21.045699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.045684 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7" Nov 03 23:00:21.048039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.047984 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" event={"ID":"dadfa6d5-cd89-4810-ae1a-928eb3439204","Type":"ContainerDied","Data":"2b6e8d9e32af3214596e656fb1c20be7ec1407589cf0c332362cdae14adb1419"} Nov 03 23:00:21.048132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.048118 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg" Nov 03 23:00:21.092896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.092410 5051 scope.go:122] "RemoveContainer" containerID="764b17b24198845384470a0349475ebf57d013b14bdbe3fe78d2c34c80811bd5" Nov 03 23:00:21.112608 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.112576 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" event={"ID":"d4691626-a281-4473-bf97-7bb4684992dc","Type":"ContainerDied","Data":"a9ae56a15199c54c91b3812d03fe51168b51d900d4727ac6c90de2dc965bafae"} Nov 03 23:00:21.112856 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.112839 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq" Nov 03 23:00:21.128373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.128342 5051 scope.go:122] "RemoveContainer" containerID="698de4832b23c18e62d464992a104a7051cdcc2b89b26f956b788f0f6b9f659d" Nov 03 23:00:21.144052 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.143954 5051 scope.go:122] "RemoveContainer" containerID="46e1151a55221cb3f2328fb8c038b1f0d80151d464660f83be79a3903d22f360" Nov 03 23:00:21.169221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.169143 5051 scope.go:122] "RemoveContainer" containerID="b82dec5defa67b1510cd2343ce980e7fc78a71196c73fd5f8ac4dda15ba0a254" Nov 03 23:00:21.353103 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.353043 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:21.357254 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.357228 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a9ab23c4-29895780-zbqz4"] Nov 03 23:00:21.379214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.379178 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:21.385983 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.385960 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:21.468881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.468857 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:21.475713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.475681 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895780-9drsq"] Nov 03 23:00:21.547034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.546968 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\") pod \"244c650a-006c-424d-9378-324f831189bf\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " Nov 03 23:00:21.547203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.547117 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\" (UniqueName: \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\") pod \"244c650a-006c-424d-9378-324f831189bf\" (UID: \"244c650a-006c-424d-9378-324f831189bf\") " Nov 03 23:00:21.547203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.547141 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\") pod \"27857e16-2d24-4253-b3e0-6e02e1d6442d\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " Nov 03 23:00:21.547320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.547232 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\" (UniqueName: \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\") pod \"27857e16-2d24-4253-b3e0-6e02e1d6442d\" (UID: \"27857e16-2d24-4253-b3e0-6e02e1d6442d\") " Nov 03 23:00:21.547383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.547354 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca" pod "244c650a-006c-424d-9378-324f831189bf" (UID: "244c650a-006c-424d-9378-324f831189bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.547578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.547551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca" pod "27857e16-2d24-4253-b3e0-6e02e1d6442d" (UID: "27857e16-2d24-4253-b3e0-6e02e1d6442d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.548295 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.548278 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27857e16-2d24-4253-b3e0-6e02e1d6442d-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.548383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.548296 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c650a-006c-424d-9378-324f831189bf-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.549494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.549458 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt" pod "244c650a-006c-424d-9378-324f831189bf" (UID: "244c650a-006c-424d-9378-324f831189bf"). InnerVolumeSpecName "kube-api-access-wvblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.549591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.549535 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c" pod "27857e16-2d24-4253-b3e0-6e02e1d6442d" (UID: "27857e16-2d24-4253-b3e0-6e02e1d6442d"). InnerVolumeSpecName "kube-api-access-2q22c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.600105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.599733 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:21.638170 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:21.637981 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:00:21.652174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.650491 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q22c\" (UniqueName: \"kubernetes.io/projected/27857e16-2d24-4253-b3e0-6e02e1d6442d-kube-api-access-2q22c\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.652174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.650529 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvblt\" (UniqueName: \"kubernetes.io/projected/244c650a-006c-424d-9378-324f831189bf-kube-api-access-wvblt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:21.787703 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.787678 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:21.819043 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.818978 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:21.823226 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.823200 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-efd39a-29895780-c9kvg"] Nov 03 23:00:21.953983 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.953950 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\" (UniqueName: \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\") pod \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " Nov 03 23:00:21.954211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.954197 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\") pod \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\" (UID: \"e21513ba-a54a-4e82-afcc-8256f0ce92a8\") " Nov 03 23:00:21.954614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.954581 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca" pod "e21513ba-a54a-4e82-afcc-8256f0ce92a8" (UID: "e21513ba-a54a-4e82-afcc-8256f0ce92a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.956377 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:21.956349 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt" pod "e21513ba-a54a-4e82-afcc-8256f0ce92a8" (UID: "e21513ba-a54a-4e82-afcc-8256f0ce92a8"). InnerVolumeSpecName "kube-api-access-mzwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.045866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.045828 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:22.049841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.049814 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4a1480-29895780-gnvq7"] Nov 03 23:00:22.055484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.055450 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21513ba-a54a-4e82-afcc-8256f0ce92a8-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.055484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.055481 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwxt\" (UniqueName: \"kubernetes.io/projected/e21513ba-a54a-4e82-afcc-8256f0ce92a8-kube-api-access-mzwxt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.133281 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.133250 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" Nov 03 23:00:22.133406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.133276 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k" event={"ID":"e21513ba-a54a-4e82-afcc-8256f0ce92a8","Type":"ContainerDied","Data":"f0f7b52d30228309c757a8654268ccac1e3164103a00bff0e6cb4120d9275b94"} Nov 03 23:00:22.133406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.133332 5051 scope.go:122] "RemoveContainer" containerID="25b3f852cbff963d9d0c26d952023fae022248398b20d1d24513bd3bd627a7f6" Nov 03 23:00:22.135031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.134980 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" Nov 03 23:00:22.135117 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.135060 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs" event={"ID":"244c650a-006c-424d-9378-324f831189bf","Type":"ContainerDied","Data":"c8c54d52a75e38ecf44c2c481d8ec1e1529868721d4a4a55271bb19daac9a524"} Nov 03 23:00:22.138273 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.138249 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" event={"ID":"27857e16-2d24-4253-b3e0-6e02e1d6442d","Type":"ContainerDied","Data":"ff792ea718d806fd162e87358965b32333465d6bc1fea590bc7611eec98c8c14"} Nov 03 23:00:22.138382 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.138283 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd" Nov 03 23:00:22.150682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.150664 5051 scope.go:122] "RemoveContainer" containerID="56ffe3d9d5a96d246f65b47c11a6242f52e259dcfab9c4e17b73b67936e35d71" Nov 03 23:00:22.184652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.171616 5051 scope.go:122] "RemoveContainer" containerID="051dc1a6b7972ee4cb253eab274d8749a09f19362d4a27c73dde18278b2eadb0" Nov 03 23:00:22.450941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.450890 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" path="/var/lib/kubelet/pods/40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc/volumes" Nov 03 23:00:22.452275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.451700 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" path="/var/lib/kubelet/pods/4ec95a0c-34f7-46fc-b52e-86267cf6570f/volumes" Nov 03 23:00:22.452524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.452443 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4691626-a281-4473-bf97-7bb4684992dc" path="/var/lib/kubelet/pods/d4691626-a281-4473-bf97-7bb4684992dc/volumes" Nov 03 23:00:22.453410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.453391 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" path="/var/lib/kubelet/pods/dadfa6d5-cd89-4810-ae1a-928eb3439204/volumes" Nov 03 23:00:22.503016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.495746 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:22.591613 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.591517 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\") pod \"2d208616-230b-40b2-8a40-56887671a1d0\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " Nov 03 23:00:22.591761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.591734 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\" (UniqueName: \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\") pod \"2d208616-230b-40b2-8a40-56887671a1d0\" (UID: \"2d208616-230b-40b2-8a40-56887671a1d0\") " Nov 03 23:00:22.591944 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.591918 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca" pod "2d208616-230b-40b2-8a40-56887671a1d0" (UID: "2d208616-230b-40b2-8a40-56887671a1d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:22.592283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.592264 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d208616-230b-40b2-8a40-56887671a1d0-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.594220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.594188 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6" pod "2d208616-230b-40b2-8a40-56887671a1d0" (UID: "2d208616-230b-40b2-8a40-56887671a1d0"). InnerVolumeSpecName "kube-api-access-6vsh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.668301 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.668270 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:22.674926 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.674894 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c93e70-29895780-phlkd"] Nov 03 23:00:22.693922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.693889 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vsh6\" (UniqueName: \"kubernetes.io/projected/2d208616-230b-40b2-8a40-56887671a1d0-kube-api-access-6vsh6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:22.777981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.777943 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:22.785370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.785338 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d31f8-29895780-lkjbs"] Nov 03 23:00:22.891812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.891773 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:22.896032 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:22.895982 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eb3c33-29895780-qsk4k"] Nov 03 23:00:23.147792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:23.147713 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" Nov 03 23:00:23.147792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:23.147760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2" event={"ID":"2d208616-230b-40b2-8a40-56887671a1d0","Type":"ContainerDied","Data":"8f325fcdc5a0e3e0fa75fc10f19943a7adcbc0af86c1585c9584b76d37883b5d"} Nov 03 23:00:23.148344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:23.147828 5051 scope.go:122] "RemoveContainer" containerID="8be4bcff0489057d0e48eeddfb2c97d4f55e8b9353cb412d1820abcd740d2851" Nov 03 23:00:23.313313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:23.313277 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:23.318242 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:23.318218 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7afb1c-29895780-2rsx2"] Nov 03 23:00:24.413738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:24.413710 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244c650a-006c-424d-9378-324f831189bf" path="/var/lib/kubelet/pods/244c650a-006c-424d-9378-324f831189bf/volumes" Nov 03 23:00:24.414207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:24.414193 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" path="/var/lib/kubelet/pods/27857e16-2d24-4253-b3e0-6e02e1d6442d/volumes" Nov 03 23:00:24.414578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:24.414565 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d208616-230b-40b2-8a40-56887671a1d0" path="/var/lib/kubelet/pods/2d208616-230b-40b2-8a40-56887671a1d0/volumes" Nov 03 23:00:24.415246 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:24.415232 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" path="/var/lib/kubelet/pods/e21513ba-a54a-4e82-afcc-8256f0ce92a8/volumes" Nov 03 23:00:25.253576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:25.253540 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 23:00:25.253814 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:25.253778 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" containerID="cri-o://079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23" gracePeriod=30 Nov 03 23:00:30.548591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.548567 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 23:00:30.606928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.606901 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607055 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607093 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607132 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607246 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:30.607289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607388 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607455 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607439 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:30.607455 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607449 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607601 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607585 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\") pod \"487d8d62-e8c3-4197-a94c-75de196ef9e0\" (UID: \"487d8d62-e8c3-4197-a94c-75de196ef9e0\") " Nov 03 23:00:30.607773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607749 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:30.608018 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.607767 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608344 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608486 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608508 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608523 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608537 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487d8d62-e8c3-4197-a94c-75de196ef9e0-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.608576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.608552 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.609735 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.609708 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:00:30.609843 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.609802 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:30.610163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.610139 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "kube-api-access-qsnkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:30.610268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.610212 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token" pod "487d8d62-e8c3-4197-a94c-75de196ef9e0" (UID: "487d8d62-e8c3-4197-a94c-75de196ef9e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:30.777956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.777865 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.777956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.777914 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnkb\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-kube-api-access-qsnkb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.777956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.777931 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:30.777956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:30.777945 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487d8d62-e8c3-4197-a94c-75de196ef9e0-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:31.227894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.227858 5051 generic.go:356] "Generic (PLEG): container finished" podID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerID="079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23" exitCode=0 Nov 03 23:00:31.228130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.227951 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" Nov 03 23:00:31.228130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.227947 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerDied","Data":"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23"} Nov 03 23:00:31.228130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.228118 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl" event={"ID":"487d8d62-e8c3-4197-a94c-75de196ef9e0","Type":"ContainerDied","Data":"e0a8a2b1d778cd881103361b9644f94b04f5a589b377c1ce784a8000c19b9557"} Nov 03 23:00:31.228282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.228144 5051 scope.go:122] "RemoveContainer" containerID="079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23" Nov 03 23:00:31.262943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.262915 5051 scope.go:122] "RemoveContainer" containerID="3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144" Nov 03 23:00:31.274423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.274403 5051 scope.go:122] "RemoveContainer" containerID="079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23" Nov 03 23:00:31.274738 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:31.274715 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23\": container with ID starting with 079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23 not found: ID does not exist" containerID="079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23" Nov 03 23:00:31.274792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.274750 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23"} err="failed to get container status \"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23\": rpc error: code = NotFound desc = could not find container \"079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23\": container with ID starting with 079d474b7084d4a02b13ec7dd5c4aa118e9633d91c694dae6ff9052ee86ffa23 not found: ID does not exist" Nov 03 23:00:31.274792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.274768 5051 scope.go:122] "RemoveContainer" containerID="3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144" Nov 03 23:00:31.275032 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:31.275012 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144\": container with ID starting with 3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144 not found: ID does not exist" containerID="3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144" Nov 03 23:00:31.275097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.275040 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144"} err="failed to get container status \"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144\": rpc error: code = NotFound desc = could not find container \"3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144\": container with ID starting with 3fdb9db93a2fe7984cedb6ddea5d4de1a6108789bf4afbe5cd5166ad27b5e144 not found: ID does not exist" Nov 03 23:00:31.407237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.407201 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 23:00:31.452022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:31.451860 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-8vndl"] Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035088 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035742 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035755 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035773 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035778 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035784 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035789 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035801 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035806 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035812 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035816 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035823 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035827 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035835 5051 cpu_manager.go:414] "containerMap: removing container" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035839 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035847 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035851 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035856 5051 cpu_manager.go:414] "containerMap: removing container" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035861 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035867 5051 cpu_manager.go:414] "containerMap: removing container" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035871 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035877 5051 cpu_manager.go:414] "containerMap: removing container" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035883 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035897 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035902 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035908 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035912 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035920 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035925 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035932 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035936 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035947 5051 cpu_manager.go:414] "containerMap: removing container" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035951 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035956 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035961 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035972 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035977 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.035983 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036006 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036021 5051 cpu_manager.go:414] "containerMap: removing container" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036027 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036036 5051 cpu_manager.go:414] "containerMap: removing container" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036042 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036055 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036059 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036066 5051 cpu_manager.go:414] "containerMap: removing container" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036070 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036092 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036099 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036107 5051 cpu_manager.go:414] "containerMap: removing container" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036110 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036120 5051 cpu_manager.go:414] "containerMap: removing container" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036124 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036133 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036138 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036269 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036278 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036282 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036289 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036295 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036300 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036305 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036311 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036316 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036320 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036325 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036329 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036333 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036338 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036342 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036347 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036353 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036357 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036366 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b524507-ace4-41fc-b16d-f97b04bd220f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036371 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036376 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036381 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" containerName="istio-proxy" Nov 03 23:00:32.040076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.036386 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:00:32.044155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.043542 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.091658 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.091627 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:00:32.117664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.117621 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.117868 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.117839 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj54v\" (UniqueName: \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.136134 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.136105 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dcf5a5\"" Nov 03 23:00:32.136320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.136151 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dcf5a5-dockercfg-7svth\"" Nov 03 23:00:32.219249 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.219219 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj54v\" (UniqueName: \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.219464 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.219383 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.220089 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.220064 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.279973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.279931 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj54v\" (UniqueName: \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\") pod \"payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.383253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.383219 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:32.413640 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.413461 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487d8d62-e8c3-4197-a94c-75de196ef9e0" path="/var/lib/kubelet/pods/487d8d62-e8c3-4197-a94c-75de196ef9e0/volumes" Nov 03 23:00:32.784486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:32.784453 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:00:32.785289 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:00:32.785260 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78959007_67b8_476a_8841_edcd2ff929f8.slice/crio-dc667bfae8f439575d57dccfae8b73ff1addc5d18dc56a62c3d91f74002cc58e WatchSource:0}: Error finding container dc667bfae8f439575d57dccfae8b73ff1addc5d18dc56a62c3d91f74002cc58e: Status 404 returned error can't find the container with id dc667bfae8f439575d57dccfae8b73ff1addc5d18dc56a62c3d91f74002cc58e Nov 03 23:00:33.249575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:33.249527 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" event={"ID":"78959007-67b8-476a-8841-edcd2ff929f8","Type":"ContainerStarted","Data":"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627"} Nov 03 23:00:33.250067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:33.249608 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" event={"ID":"78959007-67b8-476a-8841-edcd2ff929f8","Type":"ContainerStarted","Data":"dc667bfae8f439575d57dccfae8b73ff1addc5d18dc56a62c3d91f74002cc58e"} Nov 03 23:00:33.250067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:33.249769 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:33.366996 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:33.366926 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" podStartSLOduration=2.366912666 podStartE2EDuration="2.366912666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:33.364241275 +0000 UTC m=+2307.101585512" watchObservedRunningTime="2026-11-03 23:00:33.366912666 +0000 UTC m=+2307.104256932" Nov 03 23:00:44.111068 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:44.111031 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl"] Nov 03 23:00:44.160209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:44.160171 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a9ab23c4-2qrfl"] Nov 03 23:00:44.264136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:44.264110 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:00:44.410893 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:44.410808 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246c92b7-3a24-482b-bc72-0f6e321fd741" path="/var/lib/kubelet/pods/246c92b7-3a24-482b-bc72-0f6e321fd741/volumes" Nov 03 23:00:49.415690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:00:49.415660 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 23:00:51.625106 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:00:51.625064 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:01:04.439649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.439609 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" podUID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" containerName="main" containerID="cri-o://90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f" gracePeriod=45 Nov 03 23:01:04.727338 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.727316 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 23:01:04.833881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.833852 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834086 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834004 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834143 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834100 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834143 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834137 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\" (UniqueName: \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834205 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834316 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834269 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:04.834316 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834306 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\") pod \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\" (UID: \"94054aa8-acd2-4351-a7fb-bf663e5c2c2f\") " Nov 03 23:01:04.834506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834400 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:04.834506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834467 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:04.835044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834938 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.835044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834961 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.835044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.834970 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.836407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.836383 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:04.836556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.836537 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:04.836602 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.836570 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc" pod "94054aa8-acd2-4351-a7fb-bf663e5c2c2f" (UID: "94054aa8-acd2-4351-a7fb-bf663e5c2c2f"). InnerVolumeSpecName "kube-api-access-62fwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:04.936414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.936383 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fwc\" (UniqueName: \"kubernetes.io/projected/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-kube-api-access-62fwc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.936414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.936412 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:04.936414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:04.936422 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94054aa8-acd2-4351-a7fb-bf663e5c2c2f-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:05.549642 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.549605 5051 generic.go:356] "Generic (PLEG): container finished" podID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" containerID="90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f" exitCode=0 Nov 03 23:01:05.550176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.549656 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" Nov 03 23:01:05.550176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.549680 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" event={"ID":"94054aa8-acd2-4351-a7fb-bf663e5c2c2f","Type":"ContainerDied","Data":"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f"} Nov 03 23:01:05.550176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.549724 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp" event={"ID":"94054aa8-acd2-4351-a7fb-bf663e5c2c2f","Type":"ContainerDied","Data":"599baa01b6cdd5c9576963801de853a6e5c3de891a6fe9a0009f1d88a135daf2"} Nov 03 23:01:05.550176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.549741 5051 scope.go:122] "RemoveContainer" containerID="90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f" Nov 03 23:01:05.561270 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.561255 5051 scope.go:122] "RemoveContainer" containerID="90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f" Nov 03 23:01:05.561553 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:05.561535 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f\": container with ID starting with 90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f not found: ID does not exist" containerID="90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f" Nov 03 23:01:05.561604 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.561561 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f"} err="failed to get container status \"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f\": rpc error: code = NotFound desc = could not find container \"90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f\": container with ID starting with 90f7ea6954f6baf8a9b66560fa2145e1f0dc0dcd75c8f2b020f98f9bd3c6420f not found: ID does not exist" Nov 03 23:01:05.714765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.714731 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 23:01:05.764419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:05.764387 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a9ab23c4/e2e-lifecycle-model-a9ab23c4-kserve-84f969b665-dsmwp"] Nov 03 23:01:06.411302 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:06.411267 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" path="/var/lib/kubelet/pods/94054aa8-acd2-4351-a7fb-bf663e5c2c2f/volumes" Nov 03 23:01:10.793653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:10.793623 5051 scope.go:122] "RemoveContainer" containerID="594c9caccd756562e7dd37aae64fd69f6cf927ff1a4c322b2eea38fdfc5afff6" Nov 03 23:01:10.804881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:10.804857 5051 scope.go:122] "RemoveContainer" containerID="6ca7f6231e668a69da97697af81132e6954ddaad478102b25ed9465096ab20d5" Nov 03 23:01:11.732586 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.732554 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:01:11.733558 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733535 5051 cpu_manager.go:414] "containerMap: removing container" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:01:11.733558 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733556 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733578 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733583 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21513ba-a54a-4e82-afcc-8256f0ce92a8" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733595 5051 cpu_manager.go:414] "containerMap: removing container" podUID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" containerName="main" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733601 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" containerName="main" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733610 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733614 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733644 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:01:11.733724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733648 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733756 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d208616-230b-40b2-8a40-56887671a1d0" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733763 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c650a-006c-424d-9378-324f831189bf" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733771 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace9b8d8-a57b-4125-9b80-cb09625afe80" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733777 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f3a7a5-23e0-4c0b-b3cf-ec6165387fcc" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733787 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec95a0c-34f7-46fc-b52e-86267cf6570f" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733792 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadfa6d5-cd89-4810-ae1a-928eb3439204" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733797 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="27857e16-2d24-4253-b3e0-6e02e1d6442d" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733803 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4691626-a281-4473-bf97-7bb4684992dc" containerName="cleanup" Nov 03 23:01:11.733974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.733810 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="94054aa8-acd2-4351-a7fb-bf663e5c2c2f" containerName="main" Nov 03 23:01:11.739384 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.739367 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:11.796472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.796444 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:01:11.842847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.841630 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-78be2c-dockercfg-k7sfx\"" Nov 03 23:01:11.842847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.841734 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-78be2c\"" Nov 03 23:01:11.980980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.895533 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:11.981259 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.929189 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:01:11.999961 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:11.981775 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpvb\" (UniqueName: \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.017020 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.012539 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:01:12.017020 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.012644 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.101034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.101005 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pvf\" (UniqueName: \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.101231 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.101203 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpvb\" (UniqueName: \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.101370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.101355 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.101492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.101456 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.102079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.102055 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.162460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.162432 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpvb\" (UniqueName: \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\") pod \"payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.202719 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.202684 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.202913 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.202734 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pvf\" (UniqueName: \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.203436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.203410 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.264254 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.264186 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pvf\" (UniqueName: \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\") pod \"payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.331911 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.331885 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:12.354373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.354346 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:12.725033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.725000 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:01:12.725922 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:01:12.725897 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd3af34_c051_4761_a1df_08b17caee801.slice/crio-2f739a3a7d7b5c774fd2f486db9922e30fff4aa4b04d78a96ca8d8c76c6dbd93 WatchSource:0}: Error finding container 2f739a3a7d7b5c774fd2f486db9922e30fff4aa4b04d78a96ca8d8c76c6dbd93: Status 404 returned error can't find the container with id 2f739a3a7d7b5c774fd2f486db9922e30fff4aa4b04d78a96ca8d8c76c6dbd93 Nov 03 23:01:12.775056 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:01:12.775024 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3cb640_5fa3_4e34_998d_364aaf1947e2.slice/crio-0f83d32229ac1dae6ea5393fed5f93655a8949f5e33e16899d542e96cf10da6b WatchSource:0}: Error finding container 0f83d32229ac1dae6ea5393fed5f93655a8949f5e33e16899d542e96cf10da6b: Status 404 returned error can't find the container with id 0f83d32229ac1dae6ea5393fed5f93655a8949f5e33e16899d542e96cf10da6b Nov 03 23:01:12.775646 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:12.775544 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:01:13.654687 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.654648 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" event={"ID":"7dd3af34-c051-4761-a1df-08b17caee801","Type":"ContainerStarted","Data":"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2"} Nov 03 23:01:13.655236 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.654711 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" event={"ID":"7dd3af34-c051-4761-a1df-08b17caee801","Type":"ContainerStarted","Data":"2f739a3a7d7b5c774fd2f486db9922e30fff4aa4b04d78a96ca8d8c76c6dbd93"} Nov 03 23:01:13.655236 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.654942 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:13.656629 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.656596 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" event={"ID":"ac3cb640-5fa3-4e34-998d-364aaf1947e2","Type":"ContainerStarted","Data":"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a"} Nov 03 23:01:13.656759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.656652 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" event={"ID":"ac3cb640-5fa3-4e34-998d-364aaf1947e2","Type":"ContainerStarted","Data":"0f83d32229ac1dae6ea5393fed5f93655a8949f5e33e16899d542e96cf10da6b"} Nov 03 23:01:13.656759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.656680 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:13.776795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.776728 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" podStartSLOduration=2.776711193 podStartE2EDuration="2.776711193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:13.769681367 +0000 UTC m=+2347.507025775" watchObservedRunningTime="2026-11-03 23:01:13.776711193 +0000 UTC m=+2347.514055420" Nov 03 23:01:13.905827 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:13.905709 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" podStartSLOduration=2.905685302 podStartE2EDuration="2.905685302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:13.903230341 +0000 UTC m=+2347.640574568" watchObservedRunningTime="2026-11-03 23:01:13.905685302 +0000 UTC m=+2347.643029540" Nov 03 23:01:21.624791 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:21.624749 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:01:24.669232 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:24.669197 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:01:24.669660 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:24.669569 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:01:31.059167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:31.059129 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7"] Nov 03 23:01:31.107651 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:31.107616 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6682c402-48fw7"] Nov 03 23:01:32.411334 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.411299 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b427274-491f-4a08-9e61-406e24df1797" path="/var/lib/kubelet/pods/5b427274-491f-4a08-9e61-406e24df1797/volumes" Nov 03 23:01:32.759910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.759814 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 23:01:32.760187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.760141 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" containerID="cri-o://45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39" gracePeriod=30 Nov 03 23:01:32.765423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.765395 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 23:01:32.765661 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.765619 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" containerID="cri-o://7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd" gracePeriod=30 Nov 03 23:01:32.799227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.799196 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 23:01:32.799487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:32.799433 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" containerID="cri-o://57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243" gracePeriod=30 Nov 03 23:01:33.575906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:33.575870 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:33.576344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:33.575929 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:34.301070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:34.301031 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:34.301239 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:34.301092 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:34.428747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:34.428716 5051 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:34.428927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:34.428777 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:38.130260 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.130201 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 23:01:38.147712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.147689 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 23:01:38.176492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.176466 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 23:01:38.226083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226046 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.226083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226084 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226113 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226139 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226169 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226195 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226245 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.226353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226341 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226373 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226426 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226454 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226482 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226537 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226567 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.226694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226660 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226711 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226774 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226827 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226860 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226895 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226931 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.227138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227027 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227205 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\") pod \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\" (UID: \"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227239 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227285 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227363 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227404 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\") pod \"afb71b56-1aac-4e65-abb9-a52f7468225b\" (UID: \"afb71b56-1aac-4e65-abb9-a52f7468225b\") " Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226479 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227437 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\") pod \"ab58e595-3a1d-40d5-89b3-a965360f43f9\" (UID: \"ab58e595-3a1d-40d5-89b3-a965360f43f9\") " Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.228233 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.228256 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.226976 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227651 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.227820 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.228542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.228524 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.228942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.228715 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.229117 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.229083 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.230215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.229228 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.230215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.229448 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.230215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.229513 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.230509 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.230676 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.230746 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.230844 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231223 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231281 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231389 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.231523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231496 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.232280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231619 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "kube-api-access-bnrjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.232280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231627 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.232280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231676 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "kube-api-access-cv49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.232280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231711 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl" pod "afb71b56-1aac-4e65-abb9-a52f7468225b" (UID: "afb71b56-1aac-4e65-abb9-a52f7468225b"). InnerVolumeSpecName "kube-api-access-sgszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.232280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.231871 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket" pod "ab58e595-3a1d-40d5-89b3-a965360f43f9" (UID: "ab58e595-3a1d-40d5-89b3-a965360f43f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.232583 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.232451 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.233404 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.233384 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo" pod "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" (UID: "48ef9bf7-4ebe-4d1e-85a1-d716756f55ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.329942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.329907 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58e595-3a1d-40d5-89b3-a965360f43f9-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.329942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.329941 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.329955 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv49b\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-kube-api-access-cv49b\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.329970 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.329984 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330012 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330027 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330039 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330052 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330067 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgszl\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-kube-api-access-sgszl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330079 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrjd\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-kube-api-access-bnrjd\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330092 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330105 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330116 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330128 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330142 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330154 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330167 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330181 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb71b56-1aac-4e65-abb9-a52f7468225b-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330197 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330210 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58e595-3a1d-40d5-89b3-a965360f43f9-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330222 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330235 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330247 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.330873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.330262 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb71b56-1aac-4e65-abb9-a52f7468225b-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.871542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.871505 5051 generic.go:356] "Generic (PLEG): container finished" podID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerID="7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd" exitCode=0 Nov 03 23:01:38.871761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.871555 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" Nov 03 23:01:38.871761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.871595 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" event={"ID":"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae","Type":"ContainerDied","Data":"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd"} Nov 03 23:01:38.871761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.871660 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8" event={"ID":"48ef9bf7-4ebe-4d1e-85a1-d716756f55ae","Type":"ContainerDied","Data":"93ed9b0fd91a2509756b252656cc0ad1ad0072ac0bb6d8fd384bcfa1d2c86576"} Nov 03 23:01:38.871761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.871692 5051 scope.go:122] "RemoveContainer" containerID="7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd" Nov 03 23:01:38.873924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.873849 5051 generic.go:356] "Generic (PLEG): container finished" podID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerID="57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243" exitCode=0 Nov 03 23:01:38.874165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.874082 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" event={"ID":"afb71b56-1aac-4e65-abb9-a52f7468225b","Type":"ContainerDied","Data":"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243"} Nov 03 23:01:38.874165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.874133 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" event={"ID":"afb71b56-1aac-4e65-abb9-a52f7468225b","Type":"ContainerDied","Data":"8260c19229cf1b7ee2ee0630f2f66eb82e050d6dca9980489805f54c8e39fa17"} Nov 03 23:01:38.874954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.874930 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h" Nov 03 23:01:38.876207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.876175 5051 generic.go:356] "Generic (PLEG): container finished" podID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerID="45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39" exitCode=0 Nov 03 23:01:38.876308 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.876230 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" Nov 03 23:01:38.876380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.876328 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" event={"ID":"ab58e595-3a1d-40d5-89b3-a965360f43f9","Type":"ContainerDied","Data":"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39"} Nov 03 23:01:38.876380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.876366 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd" event={"ID":"ab58e595-3a1d-40d5-89b3-a965360f43f9","Type":"ContainerDied","Data":"382a4201def5e13b11fd5975c234718b790e8943bb39d6dd343dccd1becd6db4"} Nov 03 23:01:38.886869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.886837 5051 scope.go:122] "RemoveContainer" containerID="7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd" Nov 03 23:01:38.887180 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:38.887160 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd\": container with ID starting with 7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd not found: ID does not exist" containerID="7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd" Nov 03 23:01:38.887256 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.887186 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd"} err="failed to get container status \"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd\": rpc error: code = NotFound desc = could not find container \"7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd\": container with ID starting with 7d5e0fdc12f773fe66f80b32963caa93fe30b8e399ab06fca21f8858c6b9a4cd not found: ID does not exist" Nov 03 23:01:38.887256 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.887202 5051 scope.go:122] "RemoveContainer" containerID="57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243" Nov 03 23:01:38.897461 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.897442 5051 scope.go:122] "RemoveContainer" containerID="57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243" Nov 03 23:01:38.897757 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:38.897736 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243\": container with ID starting with 57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243 not found: ID does not exist" containerID="57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243" Nov 03 23:01:38.897831 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.897766 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243"} err="failed to get container status \"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243\": rpc error: code = NotFound desc = could not find container \"57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243\": container with ID starting with 57115659e0a7f07823c5780824a8e558649dd7c56c4b1afdd7d077c36cf38243 not found: ID does not exist" Nov 03 23:01:38.897831 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.897789 5051 scope.go:122] "RemoveContainer" containerID="45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39" Nov 03 23:01:38.908866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.908668 5051 scope.go:122] "RemoveContainer" containerID="45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39" Nov 03 23:01:38.908946 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:38.908928 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39\": container with ID starting with 45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39 not found: ID does not exist" containerID="45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39" Nov 03 23:01:38.909016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:38.908953 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39"} err="failed to get container status \"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39\": rpc error: code = NotFound desc = could not find container \"45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39\": container with ID starting with 45aa81ed9a13aa68246b6153dad199a80048dd0719a5407fed4a449821419e39 not found: ID does not exist" Nov 03 23:01:39.038238 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.038206 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 23:01:39.088150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.088061 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-8jb2h"] Nov 03 23:01:39.156212 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.155898 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 23:01:39.160439 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.160410 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-hjhmd"] Nov 03 23:01:39.268771 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.268715 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 23:01:39.272876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:39.272845 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-sn5r8"] Nov 03 23:01:40.411045 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:40.411016 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" path="/var/lib/kubelet/pods/48ef9bf7-4ebe-4d1e-85a1-d716756f55ae/volumes" Nov 03 23:01:40.411732 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:40.411711 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" path="/var/lib/kubelet/pods/ab58e595-3a1d-40d5-89b3-a965360f43f9/volumes" Nov 03 23:01:40.412302 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:01:40.412287 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" path="/var/lib/kubelet/pods/afb71b56-1aac-4e65-abb9-a52f7468225b/volumes" Nov 03 23:01:51.624693 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:01:51.624613 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:02:07.222299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:07.222270 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:02:07.252751 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:07.252722 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:02:09.230125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230087 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w"] Nov 03 23:02:09.230754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230740 5051 cpu_manager.go:414] "containerMap: removing container" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" Nov 03 23:02:09.230802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230755 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" Nov 03 23:02:09.230802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230770 5051 cpu_manager.go:414] "containerMap: removing container" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" Nov 03 23:02:09.230802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230775 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" Nov 03 23:02:09.230802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230801 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" Nov 03 23:02:09.230954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230806 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" Nov 03 23:02:09.230954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230912 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab58e595-3a1d-40d5-89b3-a965360f43f9" containerName="istio-proxy" Nov 03 23:02:09.230954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230927 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb71b56-1aac-4e65-abb9-a52f7468225b" containerName="istio-proxy" Nov 03 23:02:09.230954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.230935 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ef9bf7-4ebe-4d1e-85a1-d716756f55ae" containerName="istio-proxy" Nov 03 23:02:09.234692 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.234672 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.287364 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.287325 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:09.316774 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.316736 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w"] Nov 03 23:02:09.336228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.336197 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:02:09.369663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.369616 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.374712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.374646 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.375136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.374825 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skz2v\" (UniqueName: \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.476124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.476083 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skz2v\" (UniqueName: \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.476319 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.476188 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.476319 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.476273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.476868 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.476829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.478865 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.478844 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.537311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.537239 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skz2v\" (UniqueName: \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\") pod \"maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.549178 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.549149 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:09.933484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:09.933452 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w"] Nov 03 23:02:09.935201 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:02:09.935173 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb97730_07b6_44dc_b4f8_78aa8352bb00.slice/crio-c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d WatchSource:0}: Error finding container c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d: Status 404 returned error can't find the container with id c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d Nov 03 23:02:10.165796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:10.165765 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" event={"ID":"fbb97730-07b6-44dc-b4f8-78aa8352bb00","Type":"ContainerStarted","Data":"c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d"} Nov 03 23:02:10.909383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:10.909354 5051 scope.go:122] "RemoveContainer" containerID="aac9334f820b70c31f2e778c8cadaf9ca91f41ffa7d52d0a87915ceb2e62922b" Nov 03 23:02:10.925495 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:10.925466 5051 scope.go:122] "RemoveContainer" containerID="8f9d2ae36c709a84dc231366c8bcd6e486e99982b703b7a0b4e0685845c3548f" Nov 03 23:02:10.935740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:10.935718 5051 scope.go:122] "RemoveContainer" containerID="9e96c7f58095f1db3f81ec523e3f89f412aa355261643727179d5160c22de603" Nov 03 23:02:10.945518 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:10.945495 5051 scope.go:122] "RemoveContainer" containerID="b8f39b1bfc1f369ba9aa83cec1607ff394b196facd0c0859f335014936dcf4e0" Nov 03 23:02:11.175868 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:11.175787 5051 generic.go:356] "Generic (PLEG): container finished" podID="fbb97730-07b6-44dc-b4f8-78aa8352bb00" containerID="d03c143821746f463cccc044a95d56d7f3ac3d52ccb46d8219e7d5cd461e9d1b" exitCode=0 Nov 03 23:02:11.176065 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:11.175867 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" event={"ID":"fbb97730-07b6-44dc-b4f8-78aa8352bb00","Type":"ContainerDied","Data":"d03c143821746f463cccc044a95d56d7f3ac3d52ccb46d8219e7d5cd461e9d1b"} Nov 03 23:02:12.341030 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.341005 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:12.516924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.516836 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\") pod \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " Nov 03 23:02:12.516924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.516881 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\" (UniqueName: \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\") pod \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " Nov 03 23:02:12.517212 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.517066 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\") pod \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\" (UID: \"fbb97730-07b6-44dc-b4f8-78aa8352bb00\") " Nov 03 23:02:12.517457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.517427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca" pod "fbb97730-07b6-44dc-b4f8-78aa8352bb00" (UID: "fbb97730-07b6-44dc-b4f8-78aa8352bb00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:12.517710 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.517681 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:12.519479 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.519450 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v" pod "fbb97730-07b6-44dc-b4f8-78aa8352bb00" (UID: "fbb97730-07b6-44dc-b4f8-78aa8352bb00"). InnerVolumeSpecName "kube-api-access-skz2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:12.519719 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.519696 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca" pod "fbb97730-07b6-44dc-b4f8-78aa8352bb00" (UID: "fbb97730-07b6-44dc-b4f8-78aa8352bb00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:12.618657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.618618 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb97730-07b6-44dc-b4f8-78aa8352bb00-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:12.618657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:12.618653 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skz2v\" (UniqueName: \"kubernetes.io/projected/fbb97730-07b6-44dc-b4f8-78aa8352bb00-kube-api-access-skz2v\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:13.203577 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:13.203544 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" Nov 03 23:02:13.203577 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:13.203564 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w" event={"ID":"fbb97730-07b6-44dc-b4f8-78aa8352bb00","Type":"ContainerDied","Data":"c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d"} Nov 03 23:02:13.203833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:13.203648 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6290bfc46faf91f3fea9850fe064c0e15962a13b942c2dfde47549f7e78fd9d" Nov 03 23:02:21.624610 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:02:21.624566 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:02:22.697148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.691757 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:02:22.697148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.692824 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fbb97730-07b6-44dc-b4f8-78aa8352bb00" containerName="revoke-keys" Nov 03 23:02:22.697148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.692844 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb97730-07b6-44dc-b4f8-78aa8352bb00" containerName="revoke-keys" Nov 03 23:02:22.697148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.693051 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb97730-07b6-44dc-b4f8-78aa8352bb00" containerName="revoke-keys" Nov 03 23:02:22.698650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.698629 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768426 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768392 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768469 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768573 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768694 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768758 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768889 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.768948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.768931 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.769035 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.769019 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.769128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.769113 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7q6\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.800972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.800737 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cfe1bd-openshift-default-dockercfg-djccf\"" Nov 03 23:02:22.817511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.817463 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:02:22.870847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.870812 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7q6\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871065 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.870930 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871065 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871012 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871072 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871181 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871364 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871251 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871364 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871364 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871313 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871676 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871622 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871799 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.871908 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.871868 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.872117 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.872092 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.873592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.873564 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.874138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.874112 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.950638 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.950565 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:22.950792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:22.950684 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7q6\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\") pod \"e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:23.017769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:23.017730 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:23.485460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:23.484970 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:02:23.486617 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:02:23.486585 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fa2c2b_eb44_4c11_be65_0d0a751ac16c.slice/crio-a3d585bd3f4888c9cfff94d1aa79812a3a434b8e575df7b6ffd10c4ca4895ddb WatchSource:0}: Error finding container a3d585bd3f4888c9cfff94d1aa79812a3a434b8e575df7b6ffd10c4ca4895ddb: Status 404 returned error can't find the container with id a3d585bd3f4888c9cfff94d1aa79812a3a434b8e575df7b6ffd10c4ca4895ddb Nov 03 23:02:23.487574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:23.487536 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:23.487710 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:23.487614 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:23.487710 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:23.487660 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:24.293673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:24.293635 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerStarted","Data":"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd"} Nov 03 23:02:24.294105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:24.293689 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerStarted","Data":"a3d585bd3f4888c9cfff94d1aa79812a3a434b8e575df7b6ffd10c4ca4895ddb"} Nov 03 23:02:24.415009 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:24.414943 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podStartSLOduration=2.414929214 podStartE2EDuration="2.414929214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:24.414087568 +0000 UTC m=+2418.151431795" watchObservedRunningTime="2026-11-03 23:02:24.414929214 +0000 UTC m=+2418.152273438" Nov 03 23:02:25.018173 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:25.018144 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:25.024537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:25.024506 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:25.300885 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:25.300805 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:25.301883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:25.301863 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:02:51.624623 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:02:51.624579 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:02:51.999385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:51.999303 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6"] Nov 03 23:02:52.007622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.005446 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.063331 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.063296 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6"] Nov 03 23:02:52.105934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.105897 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:52.106116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.105961 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:02:52.126095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.126064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.126244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.126232 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n724x\" (UniqueName: \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.126296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.126285 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.227100 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.227070 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n724x\" (UniqueName: \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.227323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.227138 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.227323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.227203 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.227925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.227902 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.229957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.229936 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.287960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.287880 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n724x\" (UniqueName: \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\") pod \"maas-api-revoke-sub-accaaf7e-kqfw6\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.329785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.329754 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:52.711739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:52.711706 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6"] Nov 03 23:02:52.712820 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:02:52.712795 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1553a0d_1253_4761_8726_5f8e173ffb6a.slice/crio-530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730 WatchSource:0}: Error finding container 530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730: Status 404 returned error can't find the container with id 530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730 Nov 03 23:02:53.728082 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:53.728050 5051 generic.go:356] "Generic (PLEG): container finished" podID="f1553a0d-1253-4761-8726-5f8e173ffb6a" containerID="183f58119d21b8e3b6d6bf24293b238d747415acc75c91a2ec487290dc14388c" exitCode=0 Nov 03 23:02:53.728462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:53.728126 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" event={"ID":"f1553a0d-1253-4761-8726-5f8e173ffb6a","Type":"ContainerDied","Data":"183f58119d21b8e3b6d6bf24293b238d747415acc75c91a2ec487290dc14388c"} Nov 03 23:02:53.728462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:53.728176 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" event={"ID":"f1553a0d-1253-4761-8726-5f8e173ffb6a","Type":"ContainerStarted","Data":"530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730"} Nov 03 23:02:54.912531 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.912501 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:54.963226 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.963202 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\") pod \"f1553a0d-1253-4761-8726-5f8e173ffb6a\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " Nov 03 23:02:54.963388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.963269 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\" (UniqueName: \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\") pod \"f1553a0d-1253-4761-8726-5f8e173ffb6a\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " Nov 03 23:02:54.963388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.963315 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\") pod \"f1553a0d-1253-4761-8726-5f8e173ffb6a\" (UID: \"f1553a0d-1253-4761-8726-5f8e173ffb6a\") " Nov 03 23:02:54.963686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.963657 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca" pod "f1553a0d-1253-4761-8726-5f8e173ffb6a" (UID: "f1553a0d-1253-4761-8726-5f8e173ffb6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:54.963953 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.963937 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:54.965400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.965376 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca" pod "f1553a0d-1253-4761-8726-5f8e173ffb6a" (UID: "f1553a0d-1253-4761-8726-5f8e173ffb6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:54.965553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:54.965533 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x" pod "f1553a0d-1253-4761-8726-5f8e173ffb6a" (UID: "f1553a0d-1253-4761-8726-5f8e173ffb6a"). InnerVolumeSpecName "kube-api-access-n724x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:55.065655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:55.065585 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1553a0d-1253-4761-8726-5f8e173ffb6a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:55.065655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:55.065612 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n724x\" (UniqueName: \"kubernetes.io/projected/f1553a0d-1253-4761-8726-5f8e173ffb6a-kube-api-access-n724x\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:55.743807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:55.743775 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" Nov 03 23:02:55.744027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:55.743808 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6" event={"ID":"f1553a0d-1253-4761-8726-5f8e173ffb6a","Type":"ContainerDied","Data":"530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730"} Nov 03 23:02:55.744027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:55.743860 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="530a91b694e46f89908c0cac36701403c061b7688cf601939fb7ed9b1dd7b730" Nov 03 23:02:56.767937 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.767904 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf"] Nov 03 23:02:56.768778 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.768755 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f1553a0d-1253-4761-8726-5f8e173ffb6a" containerName="revoke-keys" Nov 03 23:02:56.768892 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.768780 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1553a0d-1253-4761-8726-5f8e173ffb6a" containerName="revoke-keys" Nov 03 23:02:56.769031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.769014 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1553a0d-1253-4761-8726-5f8e173ffb6a" containerName="revoke-keys" Nov 03 23:02:56.772656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.772637 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.789129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.789101 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqjq\" (UniqueName: \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.789483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.789454 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.789624 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.789526 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.829263 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.829232 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf"] Nov 03 23:02:56.874205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.874178 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:56.874372 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.874301 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:02:56.890844 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.890813 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqjq\" (UniqueName: \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.891048 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.891032 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.891116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.891071 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.891675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.891654 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.893620 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.893600 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:56.951888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:56.951861 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqjq\" (UniqueName: \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\") pod \"maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:57.087511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:57.087440 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:02:57.475304 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:57.475271 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf"] Nov 03 23:02:57.475958 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:02:57.475933 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a19fc3_222f_4051_8794_add6de9feb2b.slice/crio-71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd WatchSource:0}: Error finding container 71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd: Status 404 returned error can't find the container with id 71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd Nov 03 23:02:57.763774 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:57.763691 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" event={"ID":"f8a19fc3-222f-4051-8794-add6de9feb2b","Type":"ContainerStarted","Data":"71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd"} Nov 03 23:02:58.773687 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:58.773589 5051 generic.go:356] "Generic (PLEG): container finished" podID="f8a19fc3-222f-4051-8794-add6de9feb2b" containerID="3839a76b6345e0432a49cf0df12eb6cab0928cee7a748dceee01a3d4b7bbf407" exitCode=0 Nov 03 23:02:58.774082 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:58.773709 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" event={"ID":"f8a19fc3-222f-4051-8794-add6de9feb2b","Type":"ContainerDied","Data":"3839a76b6345e0432a49cf0df12eb6cab0928cee7a748dceee01a3d4b7bbf407"} Nov 03 23:02:59.950476 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:02:59.950447 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:03:00.038016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.037887 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\") pod \"f8a19fc3-222f-4051-8794-add6de9feb2b\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " Nov 03 23:03:00.038198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.038041 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\") pod \"f8a19fc3-222f-4051-8794-add6de9feb2b\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " Nov 03 23:03:00.038249 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.038199 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\" (UniqueName: \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\") pod \"f8a19fc3-222f-4051-8794-add6de9feb2b\" (UID: \"f8a19fc3-222f-4051-8794-add6de9feb2b\") " Nov 03 23:03:00.038462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.038427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca" pod "f8a19fc3-222f-4051-8794-add6de9feb2b" (UID: "f8a19fc3-222f-4051-8794-add6de9feb2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:00.038751 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.038729 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:00.040324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.040303 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca" pod "f8a19fc3-222f-4051-8794-add6de9feb2b" (UID: "f8a19fc3-222f-4051-8794-add6de9feb2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:00.040556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.040532 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq" pod "f8a19fc3-222f-4051-8794-add6de9feb2b" (UID: "f8a19fc3-222f-4051-8794-add6de9feb2b"). InnerVolumeSpecName "kube-api-access-pfqjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:00.140754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.140711 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfqjq\" (UniqueName: \"kubernetes.io/projected/f8a19fc3-222f-4051-8794-add6de9feb2b-kube-api-access-pfqjq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:00.140754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.140748 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a19fc3-222f-4051-8794-add6de9feb2b-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:00.793406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.793317 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" Nov 03 23:03:00.793406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.793354 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf" event={"ID":"f8a19fc3-222f-4051-8794-add6de9feb2b","Type":"ContainerDied","Data":"71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd"} Nov 03 23:03:00.793406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:00.793399 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71eeeda6792a6caa7f64173567e556a271821cd2679c23d30f6d21cc3d7f29fd" Nov 03 23:03:03.181547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.181516 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz"] Nov 03 23:03:03.182205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.182189 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f8a19fc3-222f-4051-8794-add6de9feb2b" containerName="revoke-keys" Nov 03 23:03:03.182268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.182206 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a19fc3-222f-4051-8794-add6de9feb2b" containerName="revoke-keys" Nov 03 23:03:03.182351 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.182342 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a19fc3-222f-4051-8794-add6de9feb2b" containerName="revoke-keys" Nov 03 23:03:03.186116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.186100 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.238607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.238384 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:03:03.243773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.243745 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz"] Nov 03 23:03:03.279349 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.279305 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.279530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.279440 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stqj\" (UniqueName: \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.279530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.279489 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.287868 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.287843 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:03.380788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.380759 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.381039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.380822 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stqj\" (UniqueName: \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.381039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.380873 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.381453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.381429 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.383484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.383459 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.442456 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.442378 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stqj\" (UniqueName: \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.501705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.501676 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:03.889914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:03.889886 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz"] Nov 03 23:03:03.891079 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:03:03.891054 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf06ada_4fb0_40c2_8252_2b1b1c7f915c.slice/crio-7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39 WatchSource:0}: Error finding container 7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39: Status 404 returned error can't find the container with id 7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39 Nov 03 23:03:04.825692 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:04.825645 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" event={"ID":"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c","Type":"ContainerStarted","Data":"4965d33c8462b840c5f55dc27d3ad92bb97f3e860dfb3f8c1ddbe6837e12b6c2"} Nov 03 23:03:04.826190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:04.825711 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" event={"ID":"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c","Type":"ContainerStarted","Data":"7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39"} Nov 03 23:03:04.967012 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:04.963802 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" podStartSLOduration=1.963774403 podStartE2EDuration="1.963774403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:04.947842485 +0000 UTC m=+2458.685186714" watchObservedRunningTime="2026-11-03 23:03:04.963774403 +0000 UTC m=+2458.701118693" Nov 03 23:03:05.834339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:05.834298 5051 generic.go:356] "Generic (PLEG): container finished" podID="ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" containerID="4965d33c8462b840c5f55dc27d3ad92bb97f3e860dfb3f8c1ddbe6837e12b6c2" exitCode=0 Nov 03 23:03:05.834747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:05.834380 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" event={"ID":"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c","Type":"ContainerDied","Data":"4965d33c8462b840c5f55dc27d3ad92bb97f3e860dfb3f8c1ddbe6837e12b6c2"} Nov 03 23:03:07.004779 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.004752 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:07.135375 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.135342 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\") pod \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " Nov 03 23:03:07.135596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.135540 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\") pod \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " Nov 03 23:03:07.135596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.135574 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\" (UniqueName: \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\") pod \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\" (UID: \"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c\") " Nov 03 23:03:07.135811 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.135781 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca" pod "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" (UID: "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:07.136261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.136229 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:07.138227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.138199 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca" pod "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" (UID: "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:07.139144 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.139118 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj" pod "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" (UID: "ccf06ada-4fb0-40c2-8252-2b1b1c7f915c"). InnerVolumeSpecName "kube-api-access-9stqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:07.237641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.237606 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:07.237641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.237636 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9stqj\" (UniqueName: \"kubernetes.io/projected/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c-kube-api-access-9stqj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:07.628222 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.628188 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 23:03:07.637567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.637530 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 23:03:07.678048 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.678014 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 23:03:07.853738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.853706 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" event={"ID":"ccf06ada-4fb0-40c2-8252-2b1b1c7f915c","Type":"ContainerDied","Data":"7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39"} Nov 03 23:03:07.853920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.853761 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aed91e07875215b179098f994e0b9f9a3689c03c34aa4292a456a0105d99a39" Nov 03 23:03:07.853920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:07.853720 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz" Nov 03 23:03:11.008156 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:11.008126 5051 scope.go:122] "RemoveContainer" containerID="311940f5a2a9e38437230e2bc6f43e177df4f9608b82976f6b6191ad8a051fd1" Nov 03 23:03:11.022845 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:11.022818 5051 scope.go:122] "RemoveContainer" containerID="cae2a85df2b553630a858c06d1814d34d03352b674d7583595e2f728ea7d3b32" Nov 03 23:03:11.033252 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:11.033230 5051 scope.go:122] "RemoveContainer" containerID="6a1a5173993b0bf7d3b7cca35865771e890249e79d3c0c6c02f7ceca5cb62b47" Nov 03 23:03:14.550743 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:14.550707 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 23:03:14.663728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:14.663693 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 23:03:19.236418 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.236332 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7"] Nov 03 23:03:19.237227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.237206 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" containerName="revoke-keys" Nov 03 23:03:19.237227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.237227 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" containerName="revoke-keys" Nov 03 23:03:19.237438 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.237423 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" containerName="revoke-keys" Nov 03 23:03:19.243067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.243047 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.297197 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.297164 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj"] Nov 03 23:03:19.303133 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.303108 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.341137 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341103 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.341366 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341287 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.341446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341380 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rf97\" (UniqueName: \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.341446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341426 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.341555 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341535 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.341618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.341589 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpbm\" (UniqueName: \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.342141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.342123 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:19.342267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.342146 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:03:19.351853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.351827 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7"] Nov 03 23:03:19.365084 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.365061 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj"] Nov 03 23:03:19.442783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.442752 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpbm\" (UniqueName: \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.442967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.442809 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.442967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.442899 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.443144 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.442974 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rf97\" (UniqueName: \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.443144 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.443055 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.443227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.443205 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.443658 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.443632 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.443776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.443681 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.445501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.445474 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.445621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.445604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.503870 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.503798 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpbm\" (UniqueName: \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\") pod \"maas-api-revoke-sub-105b6995-bqfzj\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.504172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.504154 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rf97\" (UniqueName: \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\") pod \"maas-api-revoke-sub-3d6244a9-n2wn7\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.551039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.550957 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" podUID="a59f895d-e385-45ea-830d-289b0d565c31" containerName="payload-processing" containerID="cri-o://8aee8e2c60b852d72d60e744ff02d5919c385437c0bfbd3bda0c887ff2ea5092" gracePeriod=25 Nov 03 23:03:19.560803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.560776 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:19.622542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.622380 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:19.665198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.664952 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" podUID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" containerName="payload-pre-processing" containerID="cri-o://4b2db9da3d2d26840688a294a0a1046ad6a8c8dde0962f0e952b7904556a4a32" gracePeriod=25 Nov 03 23:03:19.965841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.965087 5051 generic.go:356] "Generic (PLEG): container finished" podID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" containerID="4b2db9da3d2d26840688a294a0a1046ad6a8c8dde0962f0e952b7904556a4a32" exitCode=0 Nov 03 23:03:19.965841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.965157 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" event={"ID":"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc","Type":"ContainerDied","Data":"4b2db9da3d2d26840688a294a0a1046ad6a8c8dde0962f0e952b7904556a4a32"} Nov 03 23:03:19.967525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.967280 5051 generic.go:356] "Generic (PLEG): container finished" podID="a59f895d-e385-45ea-830d-289b0d565c31" containerID="8aee8e2c60b852d72d60e744ff02d5919c385437c0bfbd3bda0c887ff2ea5092" exitCode=0 Nov 03 23:03:19.967525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.967355 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" event={"ID":"a59f895d-e385-45ea-830d-289b0d565c31","Type":"ContainerDied","Data":"8aee8e2c60b852d72d60e744ff02d5919c385437c0bfbd3bda0c887ff2ea5092"} Nov 03 23:03:19.997612 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:19.989364 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7"] Nov 03 23:03:19.997612 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:03:19.989442 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ec840e_cc52_45d9_ae0e_11d2ce8594f6.slice/crio-c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a WatchSource:0}: Error finding container c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a: Status 404 returned error can't find the container with id c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a Nov 03 23:03:20.035535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.035449 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 23:03:20.137410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.137381 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 23:03:20.166738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.166701 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\") pod \"a59f895d-e385-45ea-830d-289b0d565c31\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " Nov 03 23:03:20.166909 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.166743 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\" (UniqueName: \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\") pod \"a59f895d-e385-45ea-830d-289b0d565c31\" (UID: \"a59f895d-e385-45ea-830d-289b0d565c31\") " Nov 03 23:03:20.167148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.167124 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume" pod "a59f895d-e385-45ea-830d-289b0d565c31" (UID: "a59f895d-e385-45ea-830d-289b0d565c31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:20.167477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.167454 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59f895d-e385-45ea-830d-289b0d565c31-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:20.169621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.169584 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc" pod "a59f895d-e385-45ea-830d-289b0d565c31" (UID: "a59f895d-e385-45ea-830d-289b0d565c31"). InnerVolumeSpecName "kube-api-access-cr6mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:20.238685 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.238655 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj"] Nov 03 23:03:20.239084 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:03:20.238955 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e272156_3d03_4fe9_9035_654ec582f5a2.slice/crio-6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3 WatchSource:0}: Error finding container 6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3: Status 404 returned error can't find the container with id 6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3 Nov 03 23:03:20.268806 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.268778 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\" (UniqueName: \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\") pod \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " Nov 03 23:03:20.268947 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.268931 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\") pod \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\" (UID: \"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc\") " Nov 03 23:03:20.270099 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.269666 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6mc\" (UniqueName: \"kubernetes.io/projected/a59f895d-e385-45ea-830d-289b0d565c31-kube-api-access-cr6mc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:20.270099 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.269790 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume" pod "39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" (UID: "39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:20.271370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.271342 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5" pod "39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" (UID: "39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc"). InnerVolumeSpecName "kube-api-access-vvhj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:20.371571 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.371539 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhj5\" (UniqueName: \"kubernetes.io/projected/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-kube-api-access-vvhj5\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:20.371571 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.371569 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:20.976565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.976478 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" Nov 03 23:03:20.976565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.976516 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5" event={"ID":"39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc","Type":"ContainerDied","Data":"79c0c806af3daeec272435d5776494d256660f8c367ae5974c7212fdde5c038a"} Nov 03 23:03:20.976810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.976584 5051 scope.go:122] "RemoveContainer" containerID="4b2db9da3d2d26840688a294a0a1046ad6a8c8dde0962f0e952b7904556a4a32" Nov 03 23:03:20.978351 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.978323 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" event={"ID":"3e272156-3d03-4fe9-9035-654ec582f5a2","Type":"ContainerStarted","Data":"a1477b9ed9f4c2feec8801742699e154fcfd4182e15aa087fb717553971b3cc4"} Nov 03 23:03:20.978466 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.978367 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" event={"ID":"3e272156-3d03-4fe9-9035-654ec582f5a2","Type":"ContainerStarted","Data":"6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3"} Nov 03 23:03:20.980147 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.980127 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" Nov 03 23:03:20.980252 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.980147 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5" event={"ID":"a59f895d-e385-45ea-830d-289b0d565c31","Type":"ContainerDied","Data":"cae966b9d514ce39613a16c27100039471d888c7de58470e6f913dd912b43c38"} Nov 03 23:03:20.982128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.982107 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" event={"ID":"60ec840e-cc52-45d9-ae0e-11d2ce8594f6","Type":"ContainerStarted","Data":"e0a8118f58ece671df218c1d3066812f929216ca59a287afe14fa0f145a222b6"} Nov 03 23:03:20.982251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.982151 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" event={"ID":"60ec840e-cc52-45d9-ae0e-11d2ce8594f6","Type":"ContainerStarted","Data":"c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a"} Nov 03 23:03:20.988812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:20.988793 5051 scope.go:122] "RemoveContainer" containerID="8aee8e2c60b852d72d60e744ff02d5919c385437c0bfbd3bda0c887ff2ea5092" Nov 03 23:03:21.147894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.147845 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" podStartSLOduration=2.147831601 podStartE2EDuration="2.147831601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:21.145828927 +0000 UTC m=+2474.883173166" watchObservedRunningTime="2026-11-03 23:03:21.147831601 +0000 UTC m=+2474.885175839" Nov 03 23:03:21.261535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.261497 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 23:03:21.266057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.266033 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7afb1c-869549494c-272x5"] Nov 03 23:03:21.376053 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.376021 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 23:03:21.379762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.379739 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7afb1c-5777db7584-dvxk5"] Nov 03 23:03:21.494446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.494387 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" podStartSLOduration=2.494368958 podStartE2EDuration="2.494368958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:21.488034951 +0000 UTC m=+2475.225379177" watchObservedRunningTime="2026-11-03 23:03:21.494368958 +0000 UTC m=+2475.231713186" Nov 03 23:03:21.624488 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:03:21.624444 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:03:21.991601 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.991510 5051 generic.go:356] "Generic (PLEG): container finished" podID="60ec840e-cc52-45d9-ae0e-11d2ce8594f6" containerID="e0a8118f58ece671df218c1d3066812f929216ca59a287afe14fa0f145a222b6" exitCode=0 Nov 03 23:03:21.991601 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.991548 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" event={"ID":"60ec840e-cc52-45d9-ae0e-11d2ce8594f6","Type":"ContainerDied","Data":"e0a8118f58ece671df218c1d3066812f929216ca59a287afe14fa0f145a222b6"} Nov 03 23:03:21.994485 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.994461 5051 generic.go:356] "Generic (PLEG): container finished" podID="3e272156-3d03-4fe9-9035-654ec582f5a2" containerID="a1477b9ed9f4c2feec8801742699e154fcfd4182e15aa087fb717553971b3cc4" exitCode=0 Nov 03 23:03:21.994619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:21.994549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" event={"ID":"3e272156-3d03-4fe9-9035-654ec582f5a2","Type":"ContainerDied","Data":"a1477b9ed9f4c2feec8801742699e154fcfd4182e15aa087fb717553971b3cc4"} Nov 03 23:03:22.410641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:22.410612 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" path="/var/lib/kubelet/pods/39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc/volumes" Nov 03 23:03:22.411125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:22.411110 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59f895d-e385-45ea-830d-289b0d565c31" path="/var/lib/kubelet/pods/a59f895d-e385-45ea-830d-289b0d565c31/volumes" Nov 03 23:03:22.650941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:22.650899 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" containerID="cri-o://115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382" gracePeriod=45 Nov 03 23:03:22.660442 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:22.660405 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" containerID="cri-o://51281819dad9db1de8800a05517d5bdfca6cfa22d0838bda1138f4facf4daa43" gracePeriod=45 Nov 03 23:03:22.698971 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:22.698930 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" containerID="cri-o://c32dc26048f845f28e3ab1293c353b5d168285ee9aefb47d96477d2630c96118" gracePeriod=45 Nov 03 23:03:23.003037 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.001628 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 23:03:23.010410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.010203 5051 generic.go:356] "Generic (PLEG): container finished" podID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerID="c32dc26048f845f28e3ab1293c353b5d168285ee9aefb47d96477d2630c96118" exitCode=0 Nov 03 23:03:23.010410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.010342 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" event={"ID":"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499","Type":"ContainerDied","Data":"c32dc26048f845f28e3ab1293c353b5d168285ee9aefb47d96477d2630c96118"} Nov 03 23:03:23.012192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.012163 5051 generic.go:356] "Generic (PLEG): container finished" podID="c5ab1960-193f-449d-b053-3e559a376825" containerID="115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382" exitCode=0 Nov 03 23:03:23.012429 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.012413 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" Nov 03 23:03:23.012592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.012575 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" event={"ID":"c5ab1960-193f-449d-b053-3e559a376825","Type":"ContainerDied","Data":"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382"} Nov 03 23:03:23.012639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.012609 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w" event={"ID":"c5ab1960-193f-449d-b053-3e559a376825","Type":"ContainerDied","Data":"ae1684829915334337f364bf602ac4af0cf791a7ea99bc0b429b8a43c63f5ec6"} Nov 03 23:03:23.012639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.012631 5051 scope.go:122] "RemoveContainer" containerID="115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382" Nov 03 23:03:23.017045 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.017021 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 23:03:23.017657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.017632 5051 generic.go:356] "Generic (PLEG): container finished" podID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerID="51281819dad9db1de8800a05517d5bdfca6cfa22d0838bda1138f4facf4daa43" exitCode=0 Nov 03 23:03:23.017919 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.017896 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" event={"ID":"e16565db-7bed-4f51-acf9-84afc0d7e603","Type":"ContainerDied","Data":"51281819dad9db1de8800a05517d5bdfca6cfa22d0838bda1138f4facf4daa43"} Nov 03 23:03:23.032403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.032375 5051 scope.go:122] "RemoveContainer" containerID="115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382" Nov 03 23:03:23.032830 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:03:23.032794 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382\": container with ID starting with 115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382 not found: ID does not exist" containerID="115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382" Nov 03 23:03:23.032963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.032844 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382"} err="failed to get container status \"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382\": rpc error: code = NotFound desc = could not find container \"115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382\": container with ID starting with 115d065a6bd0644a08583f0158efa3c9e1d83325cdb7a387802ae0a602b20382 not found: ID does not exist" Nov 03 23:03:23.047357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.047289 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111670 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111745 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111836 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\" (UniqueName: \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111912 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\" (UniqueName: \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111945 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\" (UniqueName: \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.111984 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112051 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112086 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112120 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112146 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112212 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\" (UniqueName: \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112260 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\") pod \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\" (UID: \"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112284 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112313 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112345 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112380 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112468 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\") pod \"e16565db-7bed-4f51-acf9-84afc0d7e603\" (UID: \"e16565db-7bed-4f51-acf9-84afc0d7e603\") " Nov 03 23:03:23.113569 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.112513 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\") pod \"c5ab1960-193f-449d-b053-3e559a376825\" (UID: \"c5ab1960-193f-449d-b053-3e559a376825\") " Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116087 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116301 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116321 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116511 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116651 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116769 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116885 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.117320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.116910 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.123057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.120909 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.123057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.121062 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "kube-api-access-6xdd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.125080 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.125042 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.125422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.125397 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "kube-api-access-vww7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.125568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.125546 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.127388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.127359 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.130776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.129962 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "kube-api-access-7s85g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.130776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.130132 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm" pod "c5ab1960-193f-449d-b053-3e559a376825" (UID: "c5ab1960-193f-449d-b053-3e559a376825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.130776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.130137 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm" pod "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" (UID: "c3fd7064-cd70-4b2e-a9bf-71d4c4d97499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:23.133280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.133242 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs" pod "e16565db-7bed-4f51-acf9-84afc0d7e603" (UID: "e16565db-7bed-4f51-acf9-84afc0d7e603"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.203691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.203621 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:23.214716 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214685 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214720 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214736 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xdd2\" (UniqueName: \"kubernetes.io/projected/e16565db-7bed-4f51-acf9-84afc0d7e603-kube-api-access-6xdd2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214755 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s85g\" (UniqueName: \"kubernetes.io/projected/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-kube-api-access-7s85g\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214771 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vww7p\" (UniqueName: \"kubernetes.io/projected/c5ab1960-193f-449d-b053-3e559a376825-kube-api-access-vww7p\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214785 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214798 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214811 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214826 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214839 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214852 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16565db-7bed-4f51-acf9-84afc0d7e603-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214865 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.214883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214877 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.215487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214889 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.215487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214907 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.215487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214920 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ab1960-193f-449d-b053-3e559a376825-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.215487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214933 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e16565db-7bed-4f51-acf9-84afc0d7e603-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.215487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.214945 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ab1960-193f-449d-b053-3e559a376825-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.245395 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.245364 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:23.315715 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.315679 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\" (UniqueName: \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\") pod \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " Nov 03 23:03:23.315920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.315773 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\") pod \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " Nov 03 23:03:23.315920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.315904 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\") pod \"3e272156-3d03-4fe9-9035-654ec582f5a2\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " Nov 03 23:03:23.316085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.316067 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\") pod \"3e272156-3d03-4fe9-9035-654ec582f5a2\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " Nov 03 23:03:23.316192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.316179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\" (UniqueName: \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\") pod \"3e272156-3d03-4fe9-9035-654ec582f5a2\" (UID: \"3e272156-3d03-4fe9-9035-654ec582f5a2\") " Nov 03 23:03:23.316257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.316214 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\") pod \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\" (UID: \"60ec840e-cc52-45d9-ae0e-11d2ce8594f6\") " Nov 03 23:03:23.316642 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.316597 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca" pod "3e272156-3d03-4fe9-9035-654ec582f5a2" (UID: "3e272156-3d03-4fe9-9035-654ec582f5a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:23.316766 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.316731 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.317083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.317053 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca" pod "60ec840e-cc52-45d9-ae0e-11d2ce8594f6" (UID: "60ec840e-cc52-45d9-ae0e-11d2ce8594f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:23.319036 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.318943 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97" pod "60ec840e-cc52-45d9-ae0e-11d2ce8594f6" (UID: "60ec840e-cc52-45d9-ae0e-11d2ce8594f6"). InnerVolumeSpecName "kube-api-access-6rf97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.319713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.319642 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca" pod "3e272156-3d03-4fe9-9035-654ec582f5a2" (UID: "3e272156-3d03-4fe9-9035-654ec582f5a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.319943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.319916 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm" pod "3e272156-3d03-4fe9-9035-654ec582f5a2" (UID: "3e272156-3d03-4fe9-9035-654ec582f5a2"). InnerVolumeSpecName "kube-api-access-blpbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:23.320115 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.319962 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca" pod "60ec840e-cc52-45d9-ae0e-11d2ce8594f6" (UID: "60ec840e-cc52-45d9-ae0e-11d2ce8594f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.418406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.418376 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.418406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.418401 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e272156-3d03-4fe9-9035-654ec582f5a2-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.418406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.418412 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blpbm\" (UniqueName: \"kubernetes.io/projected/3e272156-3d03-4fe9-9035-654ec582f5a2-kube-api-access-blpbm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.418898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.418423 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.418898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.418433 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rf97\" (UniqueName: \"kubernetes.io/projected/60ec840e-cc52-45d9-ae0e-11d2ce8594f6-kube-api-access-6rf97\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.522752 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.522669 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 23:03:23.526673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.526632 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-w1-7afb1c-kserve-56cc87768-lq48w"] Nov 03 23:03:23.956599 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.956553 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 23:03:23.998013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.997966 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc"] Nov 03 23:03:23.998704 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998683 5051 cpu_manager.go:414] "containerMap: removing container" podUID="60ec840e-cc52-45d9-ae0e-11d2ce8594f6" containerName="revoke-keys" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998706 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ec840e-cc52-45d9-ae0e-11d2ce8594f6" containerName="revoke-keys" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998733 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998740 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998777 5051 cpu_manager.go:414] "containerMap: removing container" podUID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" containerName="payload-pre-processing" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998785 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" containerName="payload-pre-processing" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998805 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998812 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" Nov 03 23:03:23.998825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998823 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3e272156-3d03-4fe9-9035-654ec582f5a2" containerName="revoke-keys" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998831 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e272156-3d03-4fe9-9035-654ec582f5a2" containerName="revoke-keys" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998847 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a59f895d-e385-45ea-830d-289b0d565c31" containerName="payload-processing" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998854 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59f895d-e385-45ea-830d-289b0d565c31" containerName="payload-processing" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998909 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.998919 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999101 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ef8fe2-a9eb-4d08-9c02-eb2150f04ebc" containerName="payload-pre-processing" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999116 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" containerName="main" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999127 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e272156-3d03-4fe9-9035-654ec582f5a2" containerName="revoke-keys" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999137 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59f895d-e385-45ea-830d-289b0d565c31" containerName="payload-processing" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999150 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" containerName="main" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999160 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ec840e-cc52-45d9-ae0e-11d2ce8594f6" containerName="revoke-keys" Nov 03 23:03:23.999293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:23.999172 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ab1960-193f-449d-b053-3e559a376825" containerName="main" Nov 03 23:03:24.003436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.003417 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.027443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.027414 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" Nov 03 23:03:24.027443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.027438 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2" event={"ID":"e16565db-7bed-4f51-acf9-84afc0d7e603","Type":"ContainerDied","Data":"23a7945101dc31181c7d26cd097ac637f50c93cf1d37365d1cfaf9488e5dbdf0"} Nov 03 23:03:24.027672 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.027486 5051 scope.go:122] "RemoveContainer" containerID="51281819dad9db1de8800a05517d5bdfca6cfa22d0838bda1138f4facf4daa43" Nov 03 23:03:24.029302 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.029285 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" Nov 03 23:03:24.029412 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.029294 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7" event={"ID":"60ec840e-cc52-45d9-ae0e-11d2ce8594f6","Type":"ContainerDied","Data":"c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a"} Nov 03 23:03:24.029412 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.029345 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c563ebe78e0e62f7356d7f30e71b7a84adc9aaaeb12071a3f82b35faa4b2a94a" Nov 03 23:03:24.031298 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.031279 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" Nov 03 23:03:24.031419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.031292 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk" event={"ID":"c3fd7064-cd70-4b2e-a9bf-71d4c4d97499","Type":"ContainerDied","Data":"4ae2e55c96d17700346f11f47c843c5ed6a930878c8ca6e12e404f15824ef0b5"} Nov 03 23:03:24.033174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.033102 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" Nov 03 23:03:24.033174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.033120 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj" event={"ID":"3e272156-3d03-4fe9-9035-654ec582f5a2","Type":"ContainerDied","Data":"6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3"} Nov 03 23:03:24.033174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.033154 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aa9952a56f4eb10a35e7e2b2de83ebb37377d8265c5e7da5573976c491366e3" Nov 03 23:03:24.041024 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.040967 5051 scope.go:122] "RemoveContainer" containerID="c32dc26048f845f28e3ab1293c353b5d168285ee9aefb47d96477d2630c96118" Nov 03 23:03:24.061828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.061797 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc"] Nov 03 23:03:24.128454 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.128422 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.128656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.128561 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.128656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.128627 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjvt\" (UniqueName: \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.230554 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.230458 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjvt\" (UniqueName: \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.230729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.230686 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.230810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.230794 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.231507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.231432 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.233520 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.233491 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 23:03:24.233757 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.233739 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.236905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.236885 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/simulator-w1-7afb1c-kserve-57b5f79484-2g2xk"] Nov 03 23:03:24.291890 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.291852 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjvt\" (UniqueName: \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\") pod \"maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.320079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.320031 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:24.418038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.417713 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fd7064-cd70-4b2e-a9bf-71d4c4d97499" path="/var/lib/kubelet/pods/c3fd7064-cd70-4b2e-a9bf-71d4c4d97499/volumes" Nov 03 23:03:24.418283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.418265 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ab1960-193f-449d-b053-3e559a376825" path="/var/lib/kubelet/pods/c5ab1960-193f-449d-b053-3e559a376825/volumes" Nov 03 23:03:24.418609 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.418593 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 23:03:24.419014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.418622 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/premium-w1-7afb1c-kserve-75cf954585-822p2"] Nov 03 23:03:24.714120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:24.714092 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc"] Nov 03 23:03:24.714581 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:03:24.714556 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59611af9_c5ee_4069_8791_8ff054a03543.slice/crio-4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae WatchSource:0}: Error finding container 4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae: Status 404 returned error can't find the container with id 4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae Nov 03 23:03:25.045142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:25.045102 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" event={"ID":"59611af9-c5ee-4069-8791-8ff054a03543","Type":"ContainerStarted","Data":"4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae"} Nov 03 23:03:26.055634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:26.055600 5051 generic.go:356] "Generic (PLEG): container finished" podID="59611af9-c5ee-4069-8791-8ff054a03543" containerID="de42c73398b1bb37ecd247ab83b9084a3a3b1c89a045f7e4ab7de59b0500a4fa" exitCode=0 Nov 03 23:03:26.056100 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:26.055684 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" event={"ID":"59611af9-c5ee-4069-8791-8ff054a03543","Type":"ContainerDied","Data":"de42c73398b1bb37ecd247ab83b9084a3a3b1c89a045f7e4ab7de59b0500a4fa"} Nov 03 23:03:26.410464 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:26.410436 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16565db-7bed-4f51-acf9-84afc0d7e603" path="/var/lib/kubelet/pods/e16565db-7bed-4f51-acf9-84afc0d7e603/volumes" Nov 03 23:03:27.239114 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.239055 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:27.370216 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.370179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\") pod \"59611af9-c5ee-4069-8791-8ff054a03543\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " Nov 03 23:03:27.370424 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.370276 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\") pod \"59611af9-c5ee-4069-8791-8ff054a03543\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " Nov 03 23:03:27.370424 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.370322 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\" (UniqueName: \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\") pod \"59611af9-c5ee-4069-8791-8ff054a03543\" (UID: \"59611af9-c5ee-4069-8791-8ff054a03543\") " Nov 03 23:03:27.370945 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.370917 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca" pod "59611af9-c5ee-4069-8791-8ff054a03543" (UID: "59611af9-c5ee-4069-8791-8ff054a03543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:27.373471 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.373435 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca" pod "59611af9-c5ee-4069-8791-8ff054a03543" (UID: "59611af9-c5ee-4069-8791-8ff054a03543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:27.373789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.373764 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt" pod "59611af9-c5ee-4069-8791-8ff054a03543" (UID: "59611af9-c5ee-4069-8791-8ff054a03543"). InnerVolumeSpecName "kube-api-access-2gjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:27.472733 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.472692 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:27.472733 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.472731 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59611af9-c5ee-4069-8791-8ff054a03543-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:27.473056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:27.472746 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjvt\" (UniqueName: \"kubernetes.io/projected/59611af9-c5ee-4069-8791-8ff054a03543-kube-api-access-2gjvt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:28.073615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:28.073582 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" Nov 03 23:03:28.073615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:28.073604 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc" event={"ID":"59611af9-c5ee-4069-8791-8ff054a03543","Type":"ContainerDied","Data":"4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae"} Nov 03 23:03:28.073899 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:28.073665 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4522759fb57c5c84094416303f4f9053e85dbc0cadf33dd2a8d695287099ae" Nov 03 23:03:31.750830 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:31.750793 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w"] Nov 03 23:03:31.801549 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:31.801512 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea5c8f-x4s6w"] Nov 03 23:03:32.412435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:32.412405 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb97730-07b6-44dc-b4f8-78aa8352bb00" path="/var/lib/kubelet/pods/fbb97730-07b6-44dc-b4f8-78aa8352bb00/volumes" Nov 03 23:03:34.529161 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:34.529132 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 23:03:39.530033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.529956 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" podUID="c3146172-606a-4acb-a24c-8c24a85a72df" containerName="payload-processing" containerID="cri-o://e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1" gracePeriod=25 Nov 03 23:03:39.826320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.826294 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 23:03:39.864768 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.864730 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\" (UniqueName: \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\") pod \"c3146172-606a-4acb-a24c-8c24a85a72df\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " Nov 03 23:03:39.864934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.864888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\") pod \"c3146172-606a-4acb-a24c-8c24a85a72df\" (UID: \"c3146172-606a-4acb-a24c-8c24a85a72df\") " Nov 03 23:03:39.865285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.865262 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume" pod "c3146172-606a-4acb-a24c-8c24a85a72df" (UID: "c3146172-606a-4acb-a24c-8c24a85a72df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:39.865383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.865369 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3146172-606a-4acb-a24c-8c24a85a72df-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:39.867211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.867184 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k" pod "c3146172-606a-4acb-a24c-8c24a85a72df" (UID: "c3146172-606a-4acb-a24c-8c24a85a72df"). InnerVolumeSpecName "kube-api-access-rwj9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:39.967055 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:39.967016 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwj9k\" (UniqueName: \"kubernetes.io/projected/c3146172-606a-4acb-a24c-8c24a85a72df-kube-api-access-rwj9k\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:40.182009 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.181967 5051 generic.go:356] "Generic (PLEG): container finished" podID="c3146172-606a-4acb-a24c-8c24a85a72df" containerID="e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1" exitCode=0 Nov 03 23:03:40.182211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.182039 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" Nov 03 23:03:40.182211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.182056 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" event={"ID":"c3146172-606a-4acb-a24c-8c24a85a72df","Type":"ContainerDied","Data":"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1"} Nov 03 23:03:40.182211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.182117 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n" event={"ID":"c3146172-606a-4acb-a24c-8c24a85a72df","Type":"ContainerDied","Data":"3a5426c6292af35f10fb0e3d1fb55538997b7e430e48fd604117771e84788454"} Nov 03 23:03:40.182211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.182148 5051 scope.go:122] "RemoveContainer" containerID="e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1" Nov 03 23:03:40.193365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.193342 5051 scope.go:122] "RemoveContainer" containerID="e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1" Nov 03 23:03:40.193714 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:03:40.193691 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1\": container with ID starting with e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1 not found: ID does not exist" containerID="e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1" Nov 03 23:03:40.193795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.193721 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1"} err="failed to get container status \"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1\": rpc error: code = NotFound desc = could not find container \"e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1\": container with ID starting with e1a5207c448c79754d1d3a3ff55ff3466a6e0ca232757659478f03e3f4e942c1 not found: ID does not exist" Nov 03 23:03:40.348596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.348552 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 23:03:40.397392 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.397354 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f8cd0c-6b7c848d84-j265n"] Nov 03 23:03:40.410490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.410461 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3146172-606a-4acb-a24c-8c24a85a72df" path="/var/lib/kubelet/pods/c3146172-606a-4acb-a24c-8c24a85a72df/volumes" Nov 03 23:03:40.649904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.649874 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25"] Nov 03 23:03:40.650529 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650514 5051 cpu_manager.go:414] "containerMap: removing container" podUID="59611af9-c5ee-4069-8791-8ff054a03543" containerName="revoke-keys" Nov 03 23:03:40.650595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650530 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59611af9-c5ee-4069-8791-8ff054a03543" containerName="revoke-keys" Nov 03 23:03:40.650595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650561 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c3146172-606a-4acb-a24c-8c24a85a72df" containerName="payload-processing" Nov 03 23:03:40.650595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650566 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3146172-606a-4acb-a24c-8c24a85a72df" containerName="payload-processing" Nov 03 23:03:40.650697 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650653 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3146172-606a-4acb-a24c-8c24a85a72df" containerName="payload-processing" Nov 03 23:03:40.650697 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.650668 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="59611af9-c5ee-4069-8791-8ff054a03543" containerName="revoke-keys" Nov 03 23:03:40.655450 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.655432 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.707593 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.707557 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:40.707861 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.707836 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:03:40.711900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.711876 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25"] Nov 03 23:03:40.778061 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.778029 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.778244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.778083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rgw\" (UniqueName: \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.778244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.778215 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.879537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.879493 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rgw\" (UniqueName: \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.879728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.879559 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.879803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.879786 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.880285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.880259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.882176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.882158 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.940300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.940234 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rgw\" (UniqueName: \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\") pod \"maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:40.969876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:40.969847 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:41.357667 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:41.357637 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25"] Nov 03 23:03:41.359586 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:03:41.359543 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957d47cb_3134_4af4_8e4d_719d662d2f22.slice/crio-5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578 WatchSource:0}: Error finding container 5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578: Status 404 returned error can't find the container with id 5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578 Nov 03 23:03:42.202459 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:42.202421 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" event={"ID":"957d47cb-3134-4af4-8e4d-719d662d2f22","Type":"ContainerStarted","Data":"57ebf283d293da2f1307a81dff1f7d251bb159286542fd4fb92f0cbacf56873e"} Nov 03 23:03:42.202875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:42.202489 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" event={"ID":"957d47cb-3134-4af4-8e4d-719d662d2f22","Type":"ContainerStarted","Data":"5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578"} Nov 03 23:03:42.320554 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:42.320493 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" podStartSLOduration=2.320473442 podStartE2EDuration="2.320473442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:42.316662376 +0000 UTC m=+2496.054006600" watchObservedRunningTime="2026-11-03 23:03:42.320473442 +0000 UTC m=+2496.057817668" Nov 03 23:03:43.213005 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:43.212956 5051 generic.go:356] "Generic (PLEG): container finished" podID="957d47cb-3134-4af4-8e4d-719d662d2f22" containerID="57ebf283d293da2f1307a81dff1f7d251bb159286542fd4fb92f0cbacf56873e" exitCode=0 Nov 03 23:03:43.213479 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:43.213039 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" event={"ID":"957d47cb-3134-4af4-8e4d-719d662d2f22","Type":"ContainerDied","Data":"57ebf283d293da2f1307a81dff1f7d251bb159286542fd4fb92f0cbacf56873e"} Nov 03 23:03:44.387740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.387718 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:44.533638 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.533544 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\") pod \"957d47cb-3134-4af4-8e4d-719d662d2f22\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " Nov 03 23:03:44.533638 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.533631 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\") pod \"957d47cb-3134-4af4-8e4d-719d662d2f22\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " Nov 03 23:03:44.533898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.533778 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\" (UniqueName: \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\") pod \"957d47cb-3134-4af4-8e4d-719d662d2f22\" (UID: \"957d47cb-3134-4af4-8e4d-719d662d2f22\") " Nov 03 23:03:44.533965 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.533910 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca" pod "957d47cb-3134-4af4-8e4d-719d662d2f22" (UID: "957d47cb-3134-4af4-8e4d-719d662d2f22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:44.534446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.534420 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:44.536113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.536086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw" pod "957d47cb-3134-4af4-8e4d-719d662d2f22" (UID: "957d47cb-3134-4af4-8e4d-719d662d2f22"). InnerVolumeSpecName "kube-api-access-p5rgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:44.536113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.536094 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca" pod "957d47cb-3134-4af4-8e4d-719d662d2f22" (UID: "957d47cb-3134-4af4-8e4d-719d662d2f22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:44.635633 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.635598 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rgw\" (UniqueName: \"kubernetes.io/projected/957d47cb-3134-4af4-8e4d-719d662d2f22-kube-api-access-p5rgw\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:44.635633 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:44.635628 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957d47cb-3134-4af4-8e4d-719d662d2f22-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:45.102784 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:45.102749 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 23:03:45.151764 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:45.151731 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 23:03:45.241803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:45.241771 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" Nov 03 23:03:45.242059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:45.241762 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25" event={"ID":"957d47cb-3134-4af4-8e4d-719d662d2f22","Type":"ContainerDied","Data":"5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578"} Nov 03 23:03:45.242059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:45.241894 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd4c3af391b73de87232e351137c81bfc23838920fa2f77741fefa1c1137578" Nov 03 23:03:51.624025 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:03:51.623966 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:03:59.054104 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:59.054068 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8"] Nov 03 23:03:59.069189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:59.069163 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd"] Nov 03 23:03:59.103863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:59.103828 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e160a5-hn7x8"] Nov 03 23:03:59.107595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:03:59.107576 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15831477-rbptd"] Nov 03 23:04:00.125070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.125033 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" containerID="cri-o://6bb79062a8f6415256575f13b1657b72d232da8146d6223b69ce10a2b61a5e97" gracePeriod=45 Nov 03 23:04:00.176501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.176470 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" containerID="cri-o://8e78f68df61efa25cf957a03668efa2e390dc65a54a1939505b4f049a14622ed" gracePeriod=45 Nov 03 23:04:00.193318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.193285 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 23:04:00.422256 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.420109 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f2b4f1-46a3-412b-9c54-27f3512e88b5" path="/var/lib/kubelet/pods/09f2b4f1-46a3-412b-9c54-27f3512e88b5/volumes" Nov 03 23:04:00.422256 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.420865 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac22e0f7-f884-4b17-b83c-7045027756a3" path="/var/lib/kubelet/pods/ac22e0f7-f884-4b17-b83c-7045027756a3/volumes" Nov 03 23:04:00.491011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.485222 5051 generic.go:356] "Generic (PLEG): container finished" podID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerID="6bb79062a8f6415256575f13b1657b72d232da8146d6223b69ce10a2b61a5e97" exitCode=0 Nov 03 23:04:00.491011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.485427 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" event={"ID":"94c7597d-73bd-4bdf-8726-f464f6cd51b0","Type":"ContainerDied","Data":"6bb79062a8f6415256575f13b1657b72d232da8146d6223b69ce10a2b61a5e97"} Nov 03 23:04:00.491011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.488346 5051 generic.go:356] "Generic (PLEG): container finished" podID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerID="8e78f68df61efa25cf957a03668efa2e390dc65a54a1939505b4f049a14622ed" exitCode=0 Nov 03 23:04:00.491011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.488411 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" event={"ID":"bd0f9eb9-4ea5-4332-868e-9361e9e4448a","Type":"ContainerDied","Data":"8e78f68df61efa25cf957a03668efa2e390dc65a54a1939505b4f049a14622ed"} Nov 03 23:04:00.569021 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.568959 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 23:04:00.594305 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.594278 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 23:04:00.687562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687475 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.687562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687542 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.687762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687571 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.687762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687621 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.687762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687699 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\" (UniqueName: \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.687762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687739 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\" (UniqueName: \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.687905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687778 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.687905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687805 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.687905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687815 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.687905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.687863 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.688083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688026 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.688083 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688041 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688116 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\") pod \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\" (UID: \"bd0f9eb9-4ea5-4332-868e-9361e9e4448a\") " Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688154 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688226 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\") pod \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\" (UID: \"94c7597d-73bd-4bdf-8726-f464f6cd51b0\") " Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688359 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688760 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688780 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688795 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688808 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.688821 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.690912 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "kube-api-access-m2hfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.690977 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "kube-api-access-gsqvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.691345 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.693693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.692632 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:00.695050 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.695016 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.695250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.695225 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs" pod "bd0f9eb9-4ea5-4332-868e-9361e9e4448a" (UID: "bd0f9eb9-4ea5-4332-868e-9361e9e4448a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:00.695362 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.695248 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm" pod "94c7597d-73bd-4bdf-8726-f464f6cd51b0" (UID: "94c7597d-73bd-4bdf-8726-f464f6cd51b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:00.789699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789662 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqvh\" (UniqueName: \"kubernetes.io/projected/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-kube-api-access-gsqvh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789694 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hfr\" (UniqueName: \"kubernetes.io/projected/94c7597d-73bd-4bdf-8726-f464f6cd51b0-kube-api-access-m2hfr\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789706 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789949 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789717 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789949 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789728 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f9eb9-4ea5-4332-868e-9361e9e4448a-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789949 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789739 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c7597d-73bd-4bdf-8726-f464f6cd51b0-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:00.789949 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:00.789749 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c7597d-73bd-4bdf-8726-f464f6cd51b0-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:01.500605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.500528 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" Nov 03 23:04:01.500605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.500542 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm" event={"ID":"94c7597d-73bd-4bdf-8726-f464f6cd51b0","Type":"ContainerDied","Data":"aa60d8d2d792570e601b15c2d95dab78bdf288b48143baca117de8c5f7b6767a"} Nov 03 23:04:01.500605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.500592 5051 scope.go:122] "RemoveContainer" containerID="6bb79062a8f6415256575f13b1657b72d232da8146d6223b69ce10a2b61a5e97" Nov 03 23:04:01.502598 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.502581 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" Nov 03 23:04:01.502705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.502599 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" event={"ID":"bd0f9eb9-4ea5-4332-868e-9361e9e4448a","Type":"ContainerDied","Data":"c4a01a9dc6970ebbe869763a2c9a5d51d1f2980804f707296d940da7ede93270"} Nov 03 23:04:01.510385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.510342 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 23:04:01.513528 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.513497 5051 scope.go:122] "RemoveContainer" containerID="8e78f68df61efa25cf957a03668efa2e390dc65a54a1939505b4f049a14622ed" Nov 03 23:04:01.665452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.665417 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 23:04:01.670699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.670673 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/simulator-w0-b64e2e-kserve-5784dd5bd5-nbn9l"] Nov 03 23:04:01.778707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.778610 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 23:04:01.785124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:01.785097 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b64e2e/distinct-w0-b64e2e-kserve-f8c7b8f7f-5b2bm"] Nov 03 23:04:02.410909 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:02.410877 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" path="/var/lib/kubelet/pods/94c7597d-73bd-4bdf-8726-f464f6cd51b0/volumes" Nov 03 23:04:02.411337 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:02.411323 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" path="/var/lib/kubelet/pods/bd0f9eb9-4ea5-4332-868e-9361e9e4448a/volumes" Nov 03 23:04:06.994101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994064 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp"] Nov 03 23:04:06.994905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994883 5051 cpu_manager.go:414] "containerMap: removing container" podUID="957d47cb-3134-4af4-8e4d-719d662d2f22" containerName="revoke-keys" Nov 03 23:04:06.995038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994907 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957d47cb-3134-4af4-8e4d-719d662d2f22" containerName="revoke-keys" Nov 03 23:04:06.995038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994930 5051 cpu_manager.go:414] "containerMap: removing container" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" Nov 03 23:04:06.995038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994938 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" Nov 03 23:04:06.995038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994965 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" Nov 03 23:04:06.995038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.994973 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" Nov 03 23:04:06.995324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.995176 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0f9eb9-4ea5-4332-868e-9361e9e4448a" containerName="main" Nov 03 23:04:06.995324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.995203 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="957d47cb-3134-4af4-8e4d-719d662d2f22" containerName="revoke-keys" Nov 03 23:04:06.995324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:06.995215 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c7597d-73bd-4bdf-8726-f464f6cd51b0" containerName="main" Nov 03 23:04:07.000503 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.000482 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.053649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.053613 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp"] Nov 03 23:04:07.070446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.070414 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.070659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.070634 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb688\" (UniqueName: \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.070762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.070743 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.096925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.096889 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:07.096925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.096919 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:04:07.171472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.171432 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb688\" (UniqueName: \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.171709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.171542 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.171709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.171632 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.172398 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.172376 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.174377 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.174347 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.232497 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.232466 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb688\" (UniqueName: \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\") pod \"maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.316142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.316052 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:07.700389 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:07.700362 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp"] Nov 03 23:04:07.703939 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:04:07.703903 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1442399_a55f_47c0_8c18_0f21b5577b15.slice/crio-524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813 WatchSource:0}: Error finding container 524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813: Status 404 returned error can't find the container with id 524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813 Nov 03 23:04:08.564101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:08.563947 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" event={"ID":"e1442399-a55f-47c0-8c18-0f21b5577b15","Type":"ContainerStarted","Data":"9982d4694910dbf53fb0d029de39fc00ed0ca6aebd94c583226ab665e215fd72"} Nov 03 23:04:08.564101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:08.564060 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" event={"ID":"e1442399-a55f-47c0-8c18-0f21b5577b15","Type":"ContainerStarted","Data":"524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813"} Nov 03 23:04:08.681259 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:08.681199 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" podStartSLOduration=2.681184143 podStartE2EDuration="2.681184143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:08.678853832 +0000 UTC m=+2522.416198086" watchObservedRunningTime="2026-11-03 23:04:08.681184143 +0000 UTC m=+2522.418528367" Nov 03 23:04:09.586802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:09.586755 5051 generic.go:356] "Generic (PLEG): container finished" podID="e1442399-a55f-47c0-8c18-0f21b5577b15" containerID="9982d4694910dbf53fb0d029de39fc00ed0ca6aebd94c583226ab665e215fd72" exitCode=0 Nov 03 23:04:09.587319 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:09.586813 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" event={"ID":"e1442399-a55f-47c0-8c18-0f21b5577b15","Type":"ContainerDied","Data":"9982d4694910dbf53fb0d029de39fc00ed0ca6aebd94c583226ab665e215fd72"} Nov 03 23:04:10.759420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.759393 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:10.818960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.818926 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\" (UniqueName: \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\") pod \"e1442399-a55f-47c0-8c18-0f21b5577b15\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " Nov 03 23:04:10.819155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.819088 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\") pod \"e1442399-a55f-47c0-8c18-0f21b5577b15\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " Nov 03 23:04:10.819214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.819176 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\") pod \"e1442399-a55f-47c0-8c18-0f21b5577b15\" (UID: \"e1442399-a55f-47c0-8c18-0f21b5577b15\") " Nov 03 23:04:10.819544 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.819504 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca" pod "e1442399-a55f-47c0-8c18-0f21b5577b15" (UID: "e1442399-a55f-47c0-8c18-0f21b5577b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:10.819794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.819661 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:10.821389 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.821366 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca" pod "e1442399-a55f-47c0-8c18-0f21b5577b15" (UID: "e1442399-a55f-47c0-8c18-0f21b5577b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:10.821508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.821425 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688" pod "e1442399-a55f-47c0-8c18-0f21b5577b15" (UID: "e1442399-a55f-47c0-8c18-0f21b5577b15"). InnerVolumeSpecName "kube-api-access-lb688". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:10.920866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.920833 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb688\" (UniqueName: \"kubernetes.io/projected/e1442399-a55f-47c0-8c18-0f21b5577b15-kube-api-access-lb688\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:10.920866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:10.920862 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1442399-a55f-47c0-8c18-0f21b5577b15-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:11.071541 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:11.071502 5051 scope.go:122] "RemoveContainer" containerID="d0143c2bb1a42b3a6ade86ac303ca38c90032832c190544732c9c6934d3b408c" Nov 03 23:04:11.082877 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:11.082854 5051 scope.go:122] "RemoveContainer" containerID="c22a69f4d0681b3aebfb36a62b2fa6fc4c917585ee4a9df9b3c20ee59572ab4e" Nov 03 23:04:11.603127 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:11.603100 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" Nov 03 23:04:11.603348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:11.603140 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp" event={"ID":"e1442399-a55f-47c0-8c18-0f21b5577b15","Type":"ContainerDied","Data":"524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813"} Nov 03 23:04:11.603348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:11.603186 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524386a4c5f8a47e6761a272203c37cdca813f55b4a5fadc4d81b54b0c5b0813" Nov 03 23:04:18.489819 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.489786 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b"] Nov 03 23:04:18.492619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.490449 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e1442399-a55f-47c0-8c18-0f21b5577b15" containerName="revoke-keys" Nov 03 23:04:18.492619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.490466 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1442399-a55f-47c0-8c18-0f21b5577b15" containerName="revoke-keys" Nov 03 23:04:18.492619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.490578 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1442399-a55f-47c0-8c18-0f21b5577b15" containerName="revoke-keys" Nov 03 23:04:18.494181 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.494164 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.543406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.543366 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr"] Nov 03 23:04:18.548506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.548468 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.550930 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.550905 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b"] Nov 03 23:04:18.589776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.589749 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:04:18.590294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.590276 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:18.613435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.613404 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr"] Nov 03 23:04:18.634616 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.634575 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.634798 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.634627 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.634798 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.634760 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4227\" (UniqueName: \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.634970 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.634948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkjd\" (UniqueName: \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.635060 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.635032 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.635213 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.635195 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.736655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.736623 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkjd\" (UniqueName: \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.736850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.736678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.736850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.736768 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.736977 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.736954 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.737053 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.737039 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.737118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.737108 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4227\" (UniqueName: \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.737636 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.737602 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.737765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.737722 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.739415 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.739391 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.739647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.739623 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.800701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.800631 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4227\" (UniqueName: \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\") pod \"maas-api-revoke-sub-51105893-mlg9b\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.800701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.800649 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkjd\" (UniqueName: \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\") pod \"maas-api-revoke-sub-3f9ed975-btgxr\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:18.808472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.808449 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:18.889289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:18.889256 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:19.222757 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.222732 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b"] Nov 03 23:04:19.224446 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:04:19.224413 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107c3d23_7080_4bb7_9c4f_728be09bd65a.slice/crio-794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a WatchSource:0}: Error finding container 794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a: Status 404 returned error can't find the container with id 794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a Nov 03 23:04:19.313672 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.313638 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr"] Nov 03 23:04:19.314956 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:04:19.314926 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7465433_bfd2_4647_bb07_ad10675cb13a.slice/crio-83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea WatchSource:0}: Error finding container 83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea: Status 404 returned error can't find the container with id 83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea Nov 03 23:04:19.448055 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.448021 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf"] Nov 03 23:04:19.451596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.451572 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-efd39a-vq8hf"] Nov 03 23:04:19.678626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.678591 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" event={"ID":"a7465433-bfd2-4647-bb07-ad10675cb13a","Type":"ContainerStarted","Data":"83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea"} Nov 03 23:04:19.680130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:19.680099 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" event={"ID":"107c3d23-7080-4bb7-9c4f-728be09bd65a","Type":"ContainerStarted","Data":"794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a"} Nov 03 23:04:20.409414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:20.409385 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a19fc3-222f-4051-8794-add6de9feb2b" path="/var/lib/kubelet/pods/f8a19fc3-222f-4051-8794-add6de9feb2b/volumes" Nov 03 23:04:20.689979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:20.689887 5051 generic.go:356] "Generic (PLEG): container finished" podID="a7465433-bfd2-4647-bb07-ad10675cb13a" containerID="56fb8a685b076828c0642a147de648f2d5311e0b371f6aa145700c60a12f220f" exitCode=0 Nov 03 23:04:20.690396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:20.689978 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" event={"ID":"a7465433-bfd2-4647-bb07-ad10675cb13a","Type":"ContainerDied","Data":"56fb8a685b076828c0642a147de648f2d5311e0b371f6aa145700c60a12f220f"} Nov 03 23:04:20.691718 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:20.691696 5051 generic.go:356] "Generic (PLEG): container finished" podID="107c3d23-7080-4bb7-9c4f-728be09bd65a" containerID="4c694f12bd4e4f80fec3ca606f54ea0e8067cc7f6d4b29496f461103fc8d9f35" exitCode=0 Nov 03 23:04:20.691838 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:20.691760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" event={"ID":"107c3d23-7080-4bb7-9c4f-728be09bd65a","Type":"ContainerDied","Data":"4c694f12bd4e4f80fec3ca606f54ea0e8067cc7f6d4b29496f461103fc8d9f35"} Nov 03 23:04:21.624867 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:04:21.624825 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:04:21.893677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.893604 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:21.924664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.924631 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\" (UniqueName: \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\") pod \"107c3d23-7080-4bb7-9c4f-728be09bd65a\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " Nov 03 23:04:21.924824 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.924689 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\") pod \"107c3d23-7080-4bb7-9c4f-728be09bd65a\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " Nov 03 23:04:21.924915 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.924877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\") pod \"107c3d23-7080-4bb7-9c4f-728be09bd65a\" (UID: \"107c3d23-7080-4bb7-9c4f-728be09bd65a\") " Nov 03 23:04:21.925286 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.925254 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca" pod "107c3d23-7080-4bb7-9c4f-728be09bd65a" (UID: "107c3d23-7080-4bb7-9c4f-728be09bd65a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:21.926154 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.925740 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:21.927167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.927141 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227" pod "107c3d23-7080-4bb7-9c4f-728be09bd65a" (UID: "107c3d23-7080-4bb7-9c4f-728be09bd65a"). InnerVolumeSpecName "kube-api-access-c4227". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:21.927382 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.927349 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca" pod "107c3d23-7080-4bb7-9c4f-728be09bd65a" (UID: "107c3d23-7080-4bb7-9c4f-728be09bd65a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:21.949883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:21.949862 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:22.027278 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.027166 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\" (UniqueName: \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\") pod \"a7465433-bfd2-4647-bb07-ad10675cb13a\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " Nov 03 23:04:22.027278 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.027239 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\") pod \"a7465433-bfd2-4647-bb07-ad10675cb13a\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " Nov 03 23:04:22.027526 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.027383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\") pod \"a7465433-bfd2-4647-bb07-ad10675cb13a\" (UID: \"a7465433-bfd2-4647-bb07-ad10675cb13a\") " Nov 03 23:04:22.028559 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.028195 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4227\" (UniqueName: \"kubernetes.io/projected/107c3d23-7080-4bb7-9c4f-728be09bd65a-kube-api-access-c4227\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:22.028559 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.028223 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107c3d23-7080-4bb7-9c4f-728be09bd65a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:22.029070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.029048 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca" pod "a7465433-bfd2-4647-bb07-ad10675cb13a" (UID: "a7465433-bfd2-4647-bb07-ad10675cb13a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:22.032587 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.032552 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca" pod "a7465433-bfd2-4647-bb07-ad10675cb13a" (UID: "a7465433-bfd2-4647-bb07-ad10675cb13a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:22.035769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.035200 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd" pod "a7465433-bfd2-4647-bb07-ad10675cb13a" (UID: "a7465433-bfd2-4647-bb07-ad10675cb13a"). InnerVolumeSpecName "kube-api-access-plkjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:22.144081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.141733 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:22.144081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.141774 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkjd\" (UniqueName: \"kubernetes.io/projected/a7465433-bfd2-4647-bb07-ad10675cb13a-kube-api-access-plkjd\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:22.144081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.141791 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7465433-bfd2-4647-bb07-ad10675cb13a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:22.709530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.709432 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" Nov 03 23:04:22.709530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.709456 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b" event={"ID":"107c3d23-7080-4bb7-9c4f-728be09bd65a","Type":"ContainerDied","Data":"794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a"} Nov 03 23:04:22.709530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.709517 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794101d11cf7576d8d9bd7d4faf934c505fa8f05947f6f6898edbef5ae111e4a" Nov 03 23:04:22.711066 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.711036 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" event={"ID":"a7465433-bfd2-4647-bb07-ad10675cb13a","Type":"ContainerDied","Data":"83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea"} Nov 03 23:04:22.711204 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.711075 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr" Nov 03 23:04:22.711204 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:22.711085 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83faa3f215ccd839bdb00a06d7c0fbca10ae3a5350e33e5fcbec69883345ecea" Nov 03 23:04:26.377619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:26.377579 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 23:04:26.378097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:26.377854 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" containerID="cri-o://10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b" gracePeriod=30 Nov 03 23:04:27.217276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.217241 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc"] Nov 03 23:04:27.217860 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.217845 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a7465433-bfd2-4647-bb07-ad10675cb13a" containerName="revoke-keys" Nov 03 23:04:27.217914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.217861 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7465433-bfd2-4647-bb07-ad10675cb13a" containerName="revoke-keys" Nov 03 23:04:27.217914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.217881 5051 cpu_manager.go:414] "containerMap: removing container" podUID="107c3d23-7080-4bb7-9c4f-728be09bd65a" containerName="revoke-keys" Nov 03 23:04:27.217914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.217887 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107c3d23-7080-4bb7-9c4f-728be09bd65a" containerName="revoke-keys" Nov 03 23:04:27.218035 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.218012 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7465433-bfd2-4647-bb07-ad10675cb13a" containerName="revoke-keys" Nov 03 23:04:27.218035 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.218022 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="107c3d23-7080-4bb7-9c4f-728be09bd65a" containerName="revoke-keys" Nov 03 23:04:27.221769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.221750 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.280695 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.280658 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:27.280695 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.280702 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:04:27.329320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.329284 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k82\" (UniqueName: \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.329512 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.329348 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.329567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.329506 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.334425 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.334400 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc"] Nov 03 23:04:27.432170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.432099 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k82\" (UniqueName: \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.432689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.432205 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.432689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.432374 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.433497 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.433459 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.443187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.443155 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.494664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.494579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k82\" (UniqueName: \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\") pod \"maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.537097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.537068 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:27.925829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:27.925803 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc"] Nov 03 23:04:27.927136 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:04:27.927112 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535fc87_c8ed_42fb_a039_240f6d19f073.slice/crio-4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861 WatchSource:0}: Error finding container 4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861: Status 404 returned error can't find the container with id 4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861 Nov 03 23:04:28.757290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:28.757195 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" event={"ID":"8535fc87-c8ed-42fb-a039-240f6d19f073","Type":"ContainerStarted","Data":"470559ae8e6cfaafb858c0c0eb09128c8d473046c149d1e071b7a6206dfad37a"} Nov 03 23:04:28.757290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:28.757254 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" event={"ID":"8535fc87-c8ed-42fb-a039-240f6d19f073","Type":"ContainerStarted","Data":"4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861"} Nov 03 23:04:28.874809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:28.874754 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" podStartSLOduration=1.874739488 podStartE2EDuration="1.874739488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:28.87315001 +0000 UTC m=+2542.610494236" watchObservedRunningTime="2026-11-03 23:04:28.874739488 +0000 UTC m=+2542.612083710" Nov 03 23:04:29.766822 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:29.766792 5051 generic.go:356] "Generic (PLEG): container finished" podID="8535fc87-c8ed-42fb-a039-240f6d19f073" containerID="470559ae8e6cfaafb858c0c0eb09128c8d473046c149d1e071b7a6206dfad37a" exitCode=0 Nov 03 23:04:29.767305 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:29.766936 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" event={"ID":"8535fc87-c8ed-42fb-a039-240f6d19f073","Type":"ContainerDied","Data":"470559ae8e6cfaafb858c0c0eb09128c8d473046c149d1e071b7a6206dfad37a"} Nov 03 23:04:30.951942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:30.951913 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:31.081850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.081764 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\" (UniqueName: \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\") pod \"8535fc87-c8ed-42fb-a039-240f6d19f073\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " Nov 03 23:04:31.082044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.082025 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\") pod \"8535fc87-c8ed-42fb-a039-240f6d19f073\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " Nov 03 23:04:31.082124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.082085 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\") pod \"8535fc87-c8ed-42fb-a039-240f6d19f073\" (UID: \"8535fc87-c8ed-42fb-a039-240f6d19f073\") " Nov 03 23:04:31.082593 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.082563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca" pod "8535fc87-c8ed-42fb-a039-240f6d19f073" (UID: "8535fc87-c8ed-42fb-a039-240f6d19f073"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:31.094154 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.094125 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca" pod "8535fc87-c8ed-42fb-a039-240f6d19f073" (UID: "8535fc87-c8ed-42fb-a039-240f6d19f073"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:31.094154 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.094135 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82" pod "8535fc87-c8ed-42fb-a039-240f6d19f073" (UID: "8535fc87-c8ed-42fb-a039-240f6d19f073"). InnerVolumeSpecName "kube-api-access-n4k82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:31.184521 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.184482 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.184521 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.184521 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8535fc87-c8ed-42fb-a039-240f6d19f073-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.184739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.184535 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k82\" (UniqueName: \"kubernetes.io/projected/8535fc87-c8ed-42fb-a039-240f6d19f073-kube-api-access-n4k82\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.651091 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.651007 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 23:04:31.783687 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.783652 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" Nov 03 23:04:31.783920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.783659 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc" event={"ID":"8535fc87-c8ed-42fb-a039-240f6d19f073","Type":"ContainerDied","Data":"4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861"} Nov 03 23:04:31.783920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.783825 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a3bf7edc4e3a07a5b1f4a2efa45d62535202401cfb6cc730b6084945cc81861" Nov 03 23:04:31.785723 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.785694 5051 generic.go:356] "Generic (PLEG): container finished" podID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerID="10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b" exitCode=0 Nov 03 23:04:31.785876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.785744 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" Nov 03 23:04:31.785876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.785771 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerDied","Data":"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b"} Nov 03 23:04:31.785876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.785829 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2" event={"ID":"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae","Type":"ContainerDied","Data":"abb1573fab9ef4e18e61b1ef459cdc275c996a168835ce0237c3cdd7ad4768b1"} Nov 03 23:04:31.785876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.785862 5051 scope.go:122] "RemoveContainer" containerID="10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792393 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792438 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792527 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792566 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792619 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792716 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792742 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792788 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.792814 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\") pod \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\" (UID: \"98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae\") " Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.793089 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.793312 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.793332 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.793392 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.793594 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.795098 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "kube-api-access-fslwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.795381 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:31.796007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.795648 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:31.796979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.796118 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:31.797141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.797120 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token" pod "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" (UID: "98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:31.802377 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.802359 5051 scope.go:122] "RemoveContainer" containerID="255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27" Nov 03 23:04:31.814398 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.814381 5051 scope.go:122] "RemoveContainer" containerID="10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b" Nov 03 23:04:31.814680 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:04:31.814657 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b\": container with ID starting with 10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b not found: ID does not exist" containerID="10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b" Nov 03 23:04:31.814773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.814694 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b"} err="failed to get container status \"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b\": rpc error: code = NotFound desc = could not find container \"10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b\": container with ID starting with 10efcdbdb829628f2ce03a74c236b15d589f843de54c446cb8a123daa96b051b not found: ID does not exist" Nov 03 23:04:31.814773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.814718 5051 scope.go:122] "RemoveContainer" containerID="255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27" Nov 03 23:04:31.814976 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:04:31.814959 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27\": container with ID starting with 255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27 not found: ID does not exist" containerID="255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27" Nov 03 23:04:31.815060 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.814982 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27"} err="failed to get container status \"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27\": rpc error: code = NotFound desc = could not find container \"255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27\": container with ID starting with 255c9fc4d47f20df22a3ff710abac86dc505588835c81bd4b3b93163656f5a27 not found: ID does not exist" Nov 03 23:04:31.895200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895137 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslwp\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-kube-api-access-fslwp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895177 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895193 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895206 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895220 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895234 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895247 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:31.895615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:31.895260 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:32.208206 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:32.208114 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 23:04:32.211954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:32.211924 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-efd39a-openshift-default-5985657f6c-fp6r2"] Nov 03 23:04:32.408856 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:32.408811 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" path="/var/lib/kubelet/pods/98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae/volumes" Nov 03 23:04:41.537637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:41.537605 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc"] Nov 03 23:04:41.586922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:41.586888 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f8cd0c-qkcnc"] Nov 03 23:04:42.409452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:42.409423 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59611af9-c5ee-4069-8791-8ff054a03543" path="/var/lib/kubelet/pods/59611af9-c5ee-4069-8791-8ff054a03543/volumes" Nov 03 23:04:46.937166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.937083 5051 generic.go:356] "Generic (PLEG): container finished" podID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerID="4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd" exitCode=0 Nov 03 23:04:46.937166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.937157 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerDied","Data":"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd"} Nov 03 23:04:46.937667 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.937645 5051 scope.go:122] "RemoveContainer" containerID="4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd" Nov 03 23:04:46.939305 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.939276 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:46.939392 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.939348 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:46.939392 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:46.939379 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:47.903739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:47.903695 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz"] Nov 03 23:04:47.907187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:47.907165 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7afb1c-nc7xz"] Nov 03 23:04:47.948047 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:47.947972 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerStarted","Data":"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769"} Nov 03 23:04:48.018396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:48.018363 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:04:48.018612 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:48.018589 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:48.018662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:48.018646 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:48.410238 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:48.410206 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf06ada-4fb0-40c2-8252-2b1b1c7f915c" path="/var/lib/kubelet/pods/ccf06ada-4fb0-40c2-8252-2b1b1c7f915c/volumes" Nov 03 23:04:49.020322 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.020050 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:49.020322 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.020126 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:49.826800 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.826766 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff"] Nov 03 23:04:49.827383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827370 5051 cpu_manager.go:414] "containerMap: removing container" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:04:49.827457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827385 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:04:49.827457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827395 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8535fc87-c8ed-42fb-a039-240f6d19f073" containerName="revoke-keys" Nov 03 23:04:49.827457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827400 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8535fc87-c8ed-42fb-a039-240f6d19f073" containerName="revoke-keys" Nov 03 23:04:49.827457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827422 5051 cpu_manager.go:414] "containerMap: removing container" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:04:49.827457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827427 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:04:49.827621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827526 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:04:49.827621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.827533 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8535fc87-c8ed-42fb-a039-240f6d19f073" containerName="revoke-keys" Nov 03 23:04:49.831318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.831302 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:49.927894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.927852 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:49.928087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.927937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:49.928087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.927978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwvh\" (UniqueName: \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:49.939228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.939199 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:49.939390 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.939324 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:04:49.943411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:49.943391 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff"] Nov 03 23:04:50.018653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.018620 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:50.018836 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.018669 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:50.029006 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.028963 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.029449 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.029048 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.029449 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.029077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwvh\" (UniqueName: \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.029637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.029616 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.031622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.031602 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.094723 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.094640 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwvh\" (UniqueName: \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\") pod \"maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.145547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.145513 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:50.593508 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:04:50.593467 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032ff11d_bc30_4307_b16c_82fcf2ed10c5.slice/crio-99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608 WatchSource:0}: Error finding container 99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608: Status 404 returned error can't find the container with id 99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608 Nov 03 23:04:50.594916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:50.594890 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff"] Nov 03 23:04:51.009681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:51.009648 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" event={"ID":"032ff11d-bc30-4307-b16c-82fcf2ed10c5","Type":"ContainerStarted","Data":"99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608"} Nov 03 23:04:51.018221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:51.018198 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:51.018311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:51.018239 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:51.139253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:51.139189 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" podStartSLOduration=2.139176994 podStartE2EDuration="2.139176994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:51.137737213 +0000 UTC m=+2564.875081438" watchObservedRunningTime="2026-11-03 23:04:51.139176994 +0000 UTC m=+2564.876521221" Nov 03 23:04:51.625436 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:04:51.625395 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:04:52.018207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:52.018113 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:52.018207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:52.018163 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:52.024055 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:52.024019 5051 generic.go:356] "Generic (PLEG): container finished" podID="032ff11d-bc30-4307-b16c-82fcf2ed10c5" containerID="38e024441a34b36eca369d4b415db9339d5d1fd06a979943797ce87b7ba81333" exitCode=0 Nov 03 23:04:52.024206 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:52.024105 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" event={"ID":"032ff11d-bc30-4307-b16c-82fcf2ed10c5","Type":"ContainerDied","Data":"38e024441a34b36eca369d4b415db9339d5d1fd06a979943797ce87b7ba81333"} Nov 03 23:04:53.018825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.018784 5051 patch_prober.go:29] interesting pod/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 23:04:53.019385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.018845 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 23:04:53.238218 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.238191 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:53.293437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.293341 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\") pod \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " Nov 03 23:04:53.293634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.293589 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\" (UniqueName: \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\") pod \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " Nov 03 23:04:53.293714 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.293637 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\") pod \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\" (UID: \"032ff11d-bc30-4307-b16c-82fcf2ed10c5\") " Nov 03 23:04:53.294101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.294071 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca" pod "032ff11d-bc30-4307-b16c-82fcf2ed10c5" (UID: "032ff11d-bc30-4307-b16c-82fcf2ed10c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:53.294279 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.294253 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:53.296224 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.296182 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca" pod "032ff11d-bc30-4307-b16c-82fcf2ed10c5" (UID: "032ff11d-bc30-4307-b16c-82fcf2ed10c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:53.296724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.296701 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh" pod "032ff11d-bc30-4307-b16c-82fcf2ed10c5" (UID: "032ff11d-bc30-4307-b16c-82fcf2ed10c5"). InnerVolumeSpecName "kube-api-access-wkwvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:53.396360 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.396311 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwvh\" (UniqueName: \"kubernetes.io/projected/032ff11d-bc30-4307-b16c-82fcf2ed10c5-kube-api-access-wkwvh\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:53.396581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:53.396566 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032ff11d-bc30-4307-b16c-82fcf2ed10c5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:54.023681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.023650 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:04:54.024103 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.024027 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:04:54.024689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.024664 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:04:54.040314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.040287 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" Nov 03 23:04:54.040469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.040327 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff" event={"ID":"032ff11d-bc30-4307-b16c-82fcf2ed10c5","Type":"ContainerDied","Data":"99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608"} Nov 03 23:04:54.040469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.040377 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99da017fdb9a7437cb09d4a18dbe0a8e08b76c29857340b802afda04dbc3f608" Nov 03 23:04:54.860869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.860828 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 23:04:54.861151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:54.861117 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" containerID="cri-o://d14f4e3653d4fa0806fbb6e7efdbc78ea58f108e38b17d46cbeb95729b389249" gracePeriod=30 Nov 03 23:04:58.588525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:58.588481 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:04:58.589068 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:04:58.588546 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:05:00.099240 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.099204 5051 generic.go:356] "Generic (PLEG): container finished" podID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerID="d14f4e3653d4fa0806fbb6e7efdbc78ea58f108e38b17d46cbeb95729b389249" exitCode=0 Nov 03 23:05:00.099637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.099437 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerDied","Data":"d14f4e3653d4fa0806fbb6e7efdbc78ea58f108e38b17d46cbeb95729b389249"} Nov 03 23:05:00.099637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.099496 5051 scope.go:122] "RemoveContainer" containerID="952b66e07e1c47c9784d3938e7f65f090ab2820e5b27b587ad3b9a48ec9b070e" Nov 03 23:05:00.132061 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.132041 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 23:05:00.288592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288557 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.288816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288654 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.288816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288703 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.288816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288745 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.288816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288774 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.289119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288827 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.289119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288881 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.289119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288908 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.289119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288941 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:00.289119 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.288958 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\") pod \"b280a08d-b941-4f52-ad36-cac35ee9afb8\" (UID: \"b280a08d-b941-4f52-ad36-cac35ee9afb8\") " Nov 03 23:05:00.289376 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.289548 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289518 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.289690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289668 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.289784 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289764 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b280a08d-b941-4f52-ad36-cac35ee9afb8-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.289784 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289778 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.289900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289787 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.289900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.289818 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.291481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.291452 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.291653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.291632 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:00.291808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.291790 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "kube-api-access-mtcv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:00.291861 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.291810 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token" pod "b280a08d-b941-4f52-ad36-cac35ee9afb8" (UID: "b280a08d-b941-4f52-ad36-cac35ee9afb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391108 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcv6\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-kube-api-access-mtcv6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391136 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391149 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391158 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b280a08d-b941-4f52-ad36-cac35ee9afb8-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391167 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.391198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:00.391175 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b280a08d-b941-4f52-ad36-cac35ee9afb8-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:01.107974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.107943 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" Nov 03 23:05:01.108494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.107985 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs" event={"ID":"b280a08d-b941-4f52-ad36-cac35ee9afb8","Type":"ContainerDied","Data":"d24fed7df2a94331d39d5dc7c34602af07ba683bc7a8be812f6578d34e20695b"} Nov 03 23:05:01.108494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.108064 5051 scope.go:122] "RemoveContainer" containerID="d14f4e3653d4fa0806fbb6e7efdbc78ea58f108e38b17d46cbeb95729b389249" Nov 03 23:05:01.396953 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.396698 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:05:01.402491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.402462 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 23:05:01.406127 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:01.406101 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-g7kbs"] Nov 03 23:05:02.410477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:02.410445 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" path="/var/lib/kubelet/pods/b280a08d-b941-4f52-ad36-cac35ee9afb8/volumes" Nov 03 23:05:06.398420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.398380 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" podUID="78959007-67b8-476a-8841-edcd2ff929f8" containerName="payload-pre-processing" containerID="cri-o://fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627" gracePeriod=25 Nov 03 23:05:06.671824 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.671800 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:05:06.695594 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.695563 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\") pod \"78959007-67b8-476a-8841-edcd2ff929f8\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " Nov 03 23:05:06.695789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.695769 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\" (UniqueName: \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\") pod \"78959007-67b8-476a-8841-edcd2ff929f8\" (UID: \"78959007-67b8-476a-8841-edcd2ff929f8\") " Nov 03 23:05:06.696010 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.695963 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume" pod "78959007-67b8-476a-8841-edcd2ff929f8" (UID: "78959007-67b8-476a-8841-edcd2ff929f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:06.696491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.696466 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78959007-67b8-476a-8841-edcd2ff929f8-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:06.698126 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.698096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v" pod "78959007-67b8-476a-8841-edcd2ff929f8" (UID: "78959007-67b8-476a-8841-edcd2ff929f8"). InnerVolumeSpecName "kube-api-access-zj54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:06.797895 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:06.797866 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj54v\" (UniqueName: \"kubernetes.io/projected/78959007-67b8-476a-8841-edcd2ff929f8-kube-api-access-zj54v\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:07.154877 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.154840 5051 generic.go:356] "Generic (PLEG): container finished" podID="78959007-67b8-476a-8841-edcd2ff929f8" containerID="fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627" exitCode=0 Nov 03 23:05:07.155142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.154885 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" event={"ID":"78959007-67b8-476a-8841-edcd2ff929f8","Type":"ContainerDied","Data":"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627"} Nov 03 23:05:07.155142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.154925 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" Nov 03 23:05:07.155142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.154970 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975" event={"ID":"78959007-67b8-476a-8841-edcd2ff929f8","Type":"ContainerDied","Data":"dc667bfae8f439575d57dccfae8b73ff1addc5d18dc56a62c3d91f74002cc58e"} Nov 03 23:05:07.155142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.155015 5051 scope.go:122] "RemoveContainer" containerID="fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627" Nov 03 23:05:07.167115 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.167086 5051 scope.go:122] "RemoveContainer" containerID="fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627" Nov 03 23:05:07.167464 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:05:07.167443 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627\": container with ID starting with fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627 not found: ID does not exist" containerID="fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627" Nov 03 23:05:07.167539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.167476 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627"} err="failed to get container status \"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627\": rpc error: code = NotFound desc = could not find container \"fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627\": container with ID starting with fb0f10e93e557a67eac5a299a878e900a9563f3e46175a79a7eff705ab0cc627 not found: ID does not exist" Nov 03 23:05:07.320536 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.320505 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:05:07.369960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:07.369925 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcf5a5-9f4df6cf5-pt975"] Nov 03 23:05:08.409422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:08.409390 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78959007-67b8-476a-8841-edcd2ff929f8" path="/var/lib/kubelet/pods/78959007-67b8-476a-8841-edcd2ff929f8/volumes" Nov 03 23:05:11.160301 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:11.160266 5051 scope.go:122] "RemoveContainer" containerID="7e111857c3c6d3b9ff2c1d8ae508b109294b462eb46e62df6fb769d1b29b0b66" Nov 03 23:05:11.171457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:11.171436 5051 scope.go:122] "RemoveContainer" containerID="358bd48f7b5e631e325b3f6c7fbe6ea02a0533919e5d9804650acbabba314428" Nov 03 23:05:11.181500 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:11.181471 5051 scope.go:122] "RemoveContainer" containerID="f87a078673a144783ac7f79805a73558087c485a23410b27160c43b7a4202a39" Nov 03 23:05:13.348407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:13.348374 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp"] Nov 03 23:05:13.399381 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:13.399348 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-924fba-8nxrp"] Nov 03 23:05:14.410488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:14.410456 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1442399-a55f-47c0-8c18-0f21b5577b15" path="/var/lib/kubelet/pods/e1442399-a55f-47c0-8c18-0f21b5577b15/volumes" Nov 03 23:05:20.977167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977132 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n"] Nov 03 23:05:20.977751 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977736 5051 cpu_manager.go:414] "containerMap: removing container" podUID="032ff11d-bc30-4307-b16c-82fcf2ed10c5" containerName="revoke-keys" Nov 03 23:05:20.977796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977753 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032ff11d-bc30-4307-b16c-82fcf2ed10c5" containerName="revoke-keys" Nov 03 23:05:20.977796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977770 5051 cpu_manager.go:414] "containerMap: removing container" podUID="78959007-67b8-476a-8841-edcd2ff929f8" containerName="payload-pre-processing" Nov 03 23:05:20.977796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977775 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78959007-67b8-476a-8841-edcd2ff929f8" containerName="payload-pre-processing" Nov 03 23:05:20.977796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977788 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977792 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977801 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977805 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977914 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="032ff11d-bc30-4307-b16c-82fcf2ed10c5" containerName="revoke-keys" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977925 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="78959007-67b8-476a-8841-edcd2ff929f8" containerName="payload-pre-processing" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977932 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977938 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b280a08d-b941-4f52-ad36-cac35ee9afb8" containerName="istio-proxy" Nov 03 23:05:20.977960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.977946 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="98725bfa-0f3c-4e0a-8ddf-3f8cb56974ae" containerName="istio-proxy" Nov 03 23:05:20.983431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:20.983410 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.038767 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.038738 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n"] Nov 03 23:05:21.083671 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.083637 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:21.083863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.083748 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:05:21.094904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.094875 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6ps\" (UniqueName: \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.095095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.095078 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.095177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.095112 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.196772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.196740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.196772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.196777 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.197093 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.196889 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6ps\" (UniqueName: \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.197613 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.197588 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.199437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.199411 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.257734 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.257669 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6ps\" (UniqueName: \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\") pod \"maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.299696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.299661 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:21.625829 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:05:21.625791 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:05:21.686191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:21.686164 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n"] Nov 03 23:05:21.687306 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:05:21.687279 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366e37ac_40f9_4029_8c65_12cdcad1c444.slice/crio-9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5 WatchSource:0}: Error finding container 9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5: Status 404 returned error can't find the container with id 9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5 Nov 03 23:05:22.328975 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.328890 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" event={"ID":"366e37ac-40f9-4029-8c65-12cdcad1c444","Type":"ContainerStarted","Data":"ec712dd108a5f3503a645f3a5281f63fd1209e54a2fb925342a24cd813737045"} Nov 03 23:05:22.328975 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.328972 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" event={"ID":"366e37ac-40f9-4029-8c65-12cdcad1c444","Type":"ContainerStarted","Data":"9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5"} Nov 03 23:05:22.443960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.443898 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" podStartSLOduration=2.44388084 podStartE2EDuration="2.44388084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:22.442954425 +0000 UTC m=+2596.180298650" watchObservedRunningTime="2026-11-03 23:05:22.44388084 +0000 UTC m=+2596.181225107" Nov 03 23:05:22.645590 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.645553 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9"] Nov 03 23:05:22.649846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.649828 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.707848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.707813 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9"] Nov 03 23:05:22.718227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.718189 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.718435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.718407 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.718614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.718580 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tn5\" (UniqueName: \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.819689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.819647 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.819901 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.819720 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.819901 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.819771 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tn5\" (UniqueName: \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.820403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.820379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.822404 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.822375 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.880550 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.880520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tn5\" (UniqueName: \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\") pod \"maas-api-revoke-keys-e2e-api-a-1048fc-bppm9\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:22.963560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:22.963477 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:23.337280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:23.337188 5051 generic.go:356] "Generic (PLEG): container finished" podID="366e37ac-40f9-4029-8c65-12cdcad1c444" containerID="ec712dd108a5f3503a645f3a5281f63fd1209e54a2fb925342a24cd813737045" exitCode=0 Nov 03 23:05:23.337675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:23.337272 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" event={"ID":"366e37ac-40f9-4029-8c65-12cdcad1c444","Type":"ContainerDied","Data":"ec712dd108a5f3503a645f3a5281f63fd1209e54a2fb925342a24cd813737045"} Nov 03 23:05:23.346087 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:05:23.346060 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5c3b12_180c_4364_817b_9a72aaf06699.slice/crio-9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af WatchSource:0}: Error finding container 9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af: Status 404 returned error can't find the container with id 9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af Nov 03 23:05:23.347029 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:23.346981 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9"] Nov 03 23:05:24.346512 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.346468 5051 generic.go:356] "Generic (PLEG): container finished" podID="0a5c3b12-180c-4364-817b-9a72aaf06699" containerID="cc1cbfa8f443bb138e888b0c428b7f617268aae2ae66ab1f7c8d1441b4b6213a" exitCode=0 Nov 03 23:05:24.346980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.346547 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" event={"ID":"0a5c3b12-180c-4364-817b-9a72aaf06699","Type":"ContainerDied","Data":"cc1cbfa8f443bb138e888b0c428b7f617268aae2ae66ab1f7c8d1441b4b6213a"} Nov 03 23:05:24.346980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.346601 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" event={"ID":"0a5c3b12-180c-4364-817b-9a72aaf06699","Type":"ContainerStarted","Data":"9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af"} Nov 03 23:05:24.507390 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.507354 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:24.644793 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.644764 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\" (UniqueName: \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\") pod \"366e37ac-40f9-4029-8c65-12cdcad1c444\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " Nov 03 23:05:24.645085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.644978 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\") pod \"366e37ac-40f9-4029-8c65-12cdcad1c444\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " Nov 03 23:05:24.645085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.645052 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\") pod \"366e37ac-40f9-4029-8c65-12cdcad1c444\" (UID: \"366e37ac-40f9-4029-8c65-12cdcad1c444\") " Nov 03 23:05:24.645513 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.645481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca" pod "366e37ac-40f9-4029-8c65-12cdcad1c444" (UID: "366e37ac-40f9-4029-8c65-12cdcad1c444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:24.645753 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.645725 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:24.647671 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.647643 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca" pod "366e37ac-40f9-4029-8c65-12cdcad1c444" (UID: "366e37ac-40f9-4029-8c65-12cdcad1c444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:24.648034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.647970 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps" pod "366e37ac-40f9-4029-8c65-12cdcad1c444" (UID: "366e37ac-40f9-4029-8c65-12cdcad1c444"). InnerVolumeSpecName "kube-api-access-cd6ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:24.747736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.747695 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366e37ac-40f9-4029-8c65-12cdcad1c444-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:24.747736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:24.747737 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6ps\" (UniqueName: \"kubernetes.io/projected/366e37ac-40f9-4029-8c65-12cdcad1c444-kube-api-access-cd6ps\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.355116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.355087 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" Nov 03 23:05:25.355512 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.355123 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n" event={"ID":"366e37ac-40f9-4029-8c65-12cdcad1c444","Type":"ContainerDied","Data":"9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5"} Nov 03 23:05:25.355512 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.355172 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec1cf0d90e5ff376b952e1b5bea386eb81fb573a9816806f94f6e7be845dde5" Nov 03 23:05:25.535728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.535697 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:25.661655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.660838 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\" (UniqueName: \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\") pod \"0a5c3b12-180c-4364-817b-9a72aaf06699\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " Nov 03 23:05:25.661655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.660921 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\") pod \"0a5c3b12-180c-4364-817b-9a72aaf06699\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " Nov 03 23:05:25.661655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.661045 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\") pod \"0a5c3b12-180c-4364-817b-9a72aaf06699\" (UID: \"0a5c3b12-180c-4364-817b-9a72aaf06699\") " Nov 03 23:05:25.661655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.661487 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca" pod "0a5c3b12-180c-4364-817b-9a72aaf06699" (UID: "0a5c3b12-180c-4364-817b-9a72aaf06699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:25.661655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.661666 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.663502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.663473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5" pod "0a5c3b12-180c-4364-817b-9a72aaf06699" (UID: "0a5c3b12-180c-4364-817b-9a72aaf06699"). InnerVolumeSpecName "kube-api-access-w6tn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:25.664395 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.664372 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca" pod "0a5c3b12-180c-4364-817b-9a72aaf06699" (UID: "0a5c3b12-180c-4364-817b-9a72aaf06699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:25.763124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.763089 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tn5\" (UniqueName: \"kubernetes.io/projected/0a5c3b12-180c-4364-817b-9a72aaf06699-kube-api-access-w6tn5\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.763124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:25.763120 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5c3b12-180c-4364-817b-9a72aaf06699-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:26.363197 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:26.363166 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" Nov 03 23:05:26.363634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:26.363202 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9" event={"ID":"0a5c3b12-180c-4364-817b-9a72aaf06699","Type":"ContainerDied","Data":"9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af"} Nov 03 23:05:26.363634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:26.363249 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8bc8618c7ca993e683a439ec593cb4375fb68e11a29a99c08f51642f0359af" Nov 03 23:05:33.533158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:33.533114 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:05:33.639074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:33.639043 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:05:34.386788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:34.386751 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25"] Nov 03 23:05:34.390296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:34.390249 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b64e2e-cth25"] Nov 03 23:05:34.408960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:34.408934 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957d47cb-3134-4af4-8e4d-719d662d2f22" path="/var/lib/kubelet/pods/957d47cb-3134-4af4-8e4d-719d662d2f22/volumes" Nov 03 23:05:38.534348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:38.534310 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" podUID="7dd3af34-c051-4761-a1df-08b17caee801" containerName="payload-processing" containerID="cri-o://f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2" gracePeriod=25 Nov 03 23:05:38.641041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:38.641004 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" podUID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" containerName="payload-pre-processing" containerID="cri-o://7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a" gracePeriod=25 Nov 03 23:05:38.888720 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:38.882650 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:05:38.974828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:38.974803 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:05:39.020984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.020454 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\") pod \"7dd3af34-c051-4761-a1df-08b17caee801\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " Nov 03 23:05:39.020984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.020566 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\" (UniqueName: \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\") pod \"7dd3af34-c051-4761-a1df-08b17caee801\" (UID: \"7dd3af34-c051-4761-a1df-08b17caee801\") " Nov 03 23:05:39.023409 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.023359 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume" pod "7dd3af34-c051-4761-a1df-08b17caee801" (UID: "7dd3af34-c051-4761-a1df-08b17caee801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:39.025305 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.025258 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf" pod "7dd3af34-c051-4761-a1df-08b17caee801" (UID: "7dd3af34-c051-4761-a1df-08b17caee801"). InnerVolumeSpecName "kube-api-access-s9pvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:39.121767 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.121725 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\") pod \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " Nov 03 23:05:39.122209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.122080 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\" (UniqueName: \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\") pod \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\" (UID: \"ac3cb640-5fa3-4e34-998d-364aaf1947e2\") " Nov 03 23:05:39.122443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.122368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume" pod "ac3cb640-5fa3-4e34-998d-364aaf1947e2" (UID: "ac3cb640-5fa3-4e34-998d-364aaf1947e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:39.122942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.122916 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3cb640-5fa3-4e34-998d-364aaf1947e2-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.123113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.122946 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd3af34-c051-4761-a1df-08b17caee801-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.123113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.122977 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9pvf\" (UniqueName: \"kubernetes.io/projected/7dd3af34-c051-4761-a1df-08b17caee801-kube-api-access-s9pvf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.124872 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.124832 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb" pod "ac3cb640-5fa3-4e34-998d-364aaf1947e2" (UID: "ac3cb640-5fa3-4e34-998d-364aaf1947e2"). InnerVolumeSpecName "kube-api-access-qzpvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:39.224634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.224602 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpvb\" (UniqueName: \"kubernetes.io/projected/ac3cb640-5fa3-4e34-998d-364aaf1947e2-kube-api-access-qzpvb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.534332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.534245 5051 generic.go:356] "Generic (PLEG): container finished" podID="7dd3af34-c051-4761-a1df-08b17caee801" containerID="f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2" exitCode=0 Nov 03 23:05:39.534332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.534298 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" Nov 03 23:05:39.534332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.534318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" event={"ID":"7dd3af34-c051-4761-a1df-08b17caee801","Type":"ContainerDied","Data":"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2"} Nov 03 23:05:39.535033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.534369 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd" event={"ID":"7dd3af34-c051-4761-a1df-08b17caee801","Type":"ContainerDied","Data":"2f739a3a7d7b5c774fd2f486db9922e30fff4aa4b04d78a96ca8d8c76c6dbd93"} Nov 03 23:05:39.535033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.534391 5051 scope.go:122] "RemoveContainer" containerID="f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2" Nov 03 23:05:39.535841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.535732 5051 generic.go:356] "Generic (PLEG): container finished" podID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" containerID="7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a" exitCode=0 Nov 03 23:05:39.535841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.535778 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" Nov 03 23:05:39.535841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.535830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" event={"ID":"ac3cb640-5fa3-4e34-998d-364aaf1947e2","Type":"ContainerDied","Data":"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a"} Nov 03 23:05:39.536077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.535870 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn" event={"ID":"ac3cb640-5fa3-4e34-998d-364aaf1947e2","Type":"ContainerDied","Data":"0f83d32229ac1dae6ea5393fed5f93655a8949f5e33e16899d542e96cf10da6b"} Nov 03 23:05:39.546741 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.546722 5051 scope.go:122] "RemoveContainer" containerID="f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2" Nov 03 23:05:39.547060 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:05:39.547041 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2\": container with ID starting with f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2 not found: ID does not exist" containerID="f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2" Nov 03 23:05:39.547120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.547068 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2"} err="failed to get container status \"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2\": rpc error: code = NotFound desc = could not find container \"f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2\": container with ID starting with f846b8607a530f1e5ff298a7421b9d88e9faa91fdfdba52a8c77941cff3af7a2 not found: ID does not exist" Nov 03 23:05:39.547120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.547085 5051 scope.go:122] "RemoveContainer" containerID="7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a" Nov 03 23:05:39.561923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.561907 5051 scope.go:122] "RemoveContainer" containerID="7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a" Nov 03 23:05:39.562263 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:05:39.562243 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a\": container with ID starting with 7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a not found: ID does not exist" containerID="7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a" Nov 03 23:05:39.562339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.562271 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a"} err="failed to get container status \"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a\": rpc error: code = NotFound desc = could not find container \"7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a\": container with ID starting with 7314681b95f21cc68dc87648744fc95f35099e11be7bc175e11f8d0ba9b1562a not found: ID does not exist" Nov 03 23:05:39.703208 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.703176 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:05:39.752346 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.752308 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-78be2c-69859d788c-vtdqn"] Nov 03 23:05:39.823170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.823081 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:05:39.829805 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:39.829778 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78be2c-598f68f6f6-bdszd"] Nov 03 23:05:40.409005 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:40.408960 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd3af34-c051-4761-a1df-08b17caee801" path="/var/lib/kubelet/pods/7dd3af34-c051-4761-a1df-08b17caee801/volumes" Nov 03 23:05:40.409433 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:40.409419 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" path="/var/lib/kubelet/pods/ac3cb640-5fa3-4e34-998d-364aaf1947e2/volumes" Nov 03 23:05:40.834730 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:40.834642 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc"] Nov 03 23:05:40.884375 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:40.884340 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e66d59-6kxbc"] Nov 03 23:05:42.410604 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:42.410570 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8535fc87-c8ed-42fb-a039-240f6d19f073" path="/var/lib/kubelet/pods/8535fc87-c8ed-42fb-a039-240f6d19f073/volumes" Nov 03 23:05:48.778014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.777957 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk"] Nov 03 23:05:48.778874 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778851 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7dd3af34-c051-4761-a1df-08b17caee801" containerName="payload-processing" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778875 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd3af34-c051-4761-a1df-08b17caee801" containerName="payload-processing" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778904 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" containerName="payload-pre-processing" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778913 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" containerName="payload-pre-processing" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778940 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0a5c3b12-180c-4364-817b-9a72aaf06699" containerName="revoke-keys" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778948 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5c3b12-180c-4364-817b-9a72aaf06699" containerName="revoke-keys" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778965 5051 cpu_manager.go:414] "containerMap: removing container" podUID="366e37ac-40f9-4029-8c65-12cdcad1c444" containerName="revoke-keys" Nov 03 23:05:48.779017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.778972 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366e37ac-40f9-4029-8c65-12cdcad1c444" containerName="revoke-keys" Nov 03 23:05:48.779408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.779154 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5c3b12-180c-4364-817b-9a72aaf06699" containerName="revoke-keys" Nov 03 23:05:48.779408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.779170 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="366e37ac-40f9-4029-8c65-12cdcad1c444" containerName="revoke-keys" Nov 03 23:05:48.779408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.779189 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd3af34-c051-4761-a1df-08b17caee801" containerName="payload-processing" Nov 03 23:05:48.779408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.779201 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3cb640-5fa3-4e34-998d-364aaf1947e2" containerName="payload-pre-processing" Nov 03 23:05:48.784870 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.784843 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.838247 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.838209 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk"] Nov 03 23:05:48.850817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.850785 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gqg\" (UniqueName: \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.850817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.850831 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.851143 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.850854 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.883112 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.883078 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:48.883112 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.883095 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:05:48.952108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.952056 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gqg\" (UniqueName: \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.952288 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.952131 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.952288 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.952153 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.952949 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.952915 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:48.954800 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:48.954782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:49.012916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:49.012873 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gqg\" (UniqueName: \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\") pod \"maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:49.101514 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:49.101421 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:49.487670 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:49.487640 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk"] Nov 03 23:05:49.490381 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:05:49.490351 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f603ec5_91b2_41b8_986c_228831367a8a.slice/crio-8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09 WatchSource:0}: Error finding container 8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09: Status 404 returned error can't find the container with id 8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09 Nov 03 23:05:49.616648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:49.616617 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" event={"ID":"1f603ec5-91b2-41b8-986c-228831367a8a","Type":"ContainerStarted","Data":"8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09"} Nov 03 23:05:50.625748 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:50.625710 5051 generic.go:356] "Generic (PLEG): container finished" podID="1f603ec5-91b2-41b8-986c-228831367a8a" containerID="38d59c50102061364273bbe3891f710a7d156786465e6f25d46979a1cba62f22" exitCode=0 Nov 03 23:05:50.626241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:50.625794 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" event={"ID":"1f603ec5-91b2-41b8-986c-228831367a8a","Type":"ContainerDied","Data":"38d59c50102061364273bbe3891f710a7d156786465e6f25d46979a1cba62f22"} Nov 03 23:05:51.623315 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:05:51.623282 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:05:51.795580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.795555 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:51.881448 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.881368 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\") pod \"1f603ec5-91b2-41b8-986c-228831367a8a\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " Nov 03 23:05:51.881663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.881545 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\" (UniqueName: \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\") pod \"1f603ec5-91b2-41b8-986c-228831367a8a\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " Nov 03 23:05:51.881663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.881649 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\") pod \"1f603ec5-91b2-41b8-986c-228831367a8a\" (UID: \"1f603ec5-91b2-41b8-986c-228831367a8a\") " Nov 03 23:05:51.881960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.881933 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca" pod "1f603ec5-91b2-41b8-986c-228831367a8a" (UID: "1f603ec5-91b2-41b8-986c-228831367a8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:51.882230 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.882204 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:51.883705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.883683 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca" pod "1f603ec5-91b2-41b8-986c-228831367a8a" (UID: "1f603ec5-91b2-41b8-986c-228831367a8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:51.883884 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.883870 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg" pod "1f603ec5-91b2-41b8-986c-228831367a8a" (UID: "1f603ec5-91b2-41b8-986c-228831367a8a"). InnerVolumeSpecName "kube-api-access-d7gqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:51.983824 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.983787 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f603ec5-91b2-41b8-986c-228831367a8a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:51.983824 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:51.983831 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7gqg\" (UniqueName: \"kubernetes.io/projected/1f603ec5-91b2-41b8-986c-228831367a8a-kube-api-access-d7gqg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:52.641922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:52.641891 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" event={"ID":"1f603ec5-91b2-41b8-986c-228831367a8a","Type":"ContainerDied","Data":"8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09"} Nov 03 23:05:52.642129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:52.641938 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8258bc82d9c71099eb1b977bfd02be6c67dd963aa1758c653fe4dfe558d22a09" Nov 03 23:05:52.642129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:52.641934 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk" Nov 03 23:05:53.236393 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:53.236363 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 23:05:53.242027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:53.241982 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 23:05:57.364132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:57.364092 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff"] Nov 03 23:05:57.413087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:57.413049 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dcf5a5-lkvff"] Nov 03 23:05:58.409260 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:58.409227 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032ff11d-bc30-4307-b16c-82fcf2ed10c5" path="/var/lib/kubelet/pods/032ff11d-bc30-4307-b16c-82fcf2ed10c5/volumes" Nov 03 23:05:58.846719 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:58.846634 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 23:05:58.846898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:05:58.846880 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" containerID="cri-o://984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a" gracePeriod=30 Nov 03 23:06:00.271136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:00.271102 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 23:06:00.382359 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:00.382321 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 23:06:04.160963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.160925 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 23:06:04.263432 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.263396 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.264927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.264903 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.265623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.265603 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.270565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.270549 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.270936 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.270923 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.271147 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271133 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.271284 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271275 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.271568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.270346 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "kube-api-access-rf54h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:04.271568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.270483 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:04.271568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271082 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.271568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271151 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.271772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271761 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.271855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271832 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.271945 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.271934 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\") pod \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\" (UID: \"0908daf7-a76e-463d-bfc7-7a7ea80806fa\") " Nov 03 23:06:04.272414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272356 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272424 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf54h\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-kube-api-access-rf54h\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272441 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272450 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272458 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272466 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.272527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.272474 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.274200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.274177 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:04.274318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.274196 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:04.274318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.274224 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token" pod "0908daf7-a76e-463d-bfc7-7a7ea80806fa" (UID: "0908daf7-a76e-463d-bfc7-7a7ea80806fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:04.373850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.373774 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.373850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.373813 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.373850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.373829 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0908daf7-a76e-463d-bfc7-7a7ea80806fa-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:04.765910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.765827 5051 generic.go:356] "Generic (PLEG): container finished" podID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerID="984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a" exitCode=0 Nov 03 23:06:04.765910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.765869 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" Nov 03 23:06:04.766172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.765906 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" event={"ID":"0908daf7-a76e-463d-bfc7-7a7ea80806fa","Type":"ContainerDied","Data":"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a"} Nov 03 23:06:04.766172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.765960 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp" event={"ID":"0908daf7-a76e-463d-bfc7-7a7ea80806fa","Type":"ContainerDied","Data":"0b057b93b8ecc3c126ca09f01d61f46b9f5c89492c3ab24bb9e7a140953b5001"} Nov 03 23:06:04.766172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.765984 5051 scope.go:122] "RemoveContainer" containerID="984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a" Nov 03 23:06:04.776383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.776362 5051 scope.go:122] "RemoveContainer" containerID="984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a" Nov 03 23:06:04.776691 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:04.776667 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a\": container with ID starting with 984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a not found: ID does not exist" containerID="984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a" Nov 03 23:06:04.776816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.776701 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a"} err="failed to get container status \"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a\": rpc error: code = NotFound desc = could not find container \"984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a\": container with ID starting with 984f0f88b44102a4fe7f8986599f703259fdcc03d5a4535dd3f8049b0802888a not found: ID does not exist" Nov 03 23:06:04.934511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.934479 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 23:06:04.940039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.940016 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-9qxhp"] Nov 03 23:06:04.990053 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990017 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf"] Nov 03 23:06:04.990637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990622 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" Nov 03 23:06:04.990682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990640 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" Nov 03 23:06:04.990682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990656 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1f603ec5-91b2-41b8-986c-228831367a8a" containerName="revoke-keys" Nov 03 23:06:04.990682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990662 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f603ec5-91b2-41b8-986c-228831367a8a" containerName="revoke-keys" Nov 03 23:06:04.990783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990743 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" containerName="istio-proxy" Nov 03 23:06:04.990783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.990753 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f603ec5-91b2-41b8-986c-228831367a8a" containerName="revoke-keys" Nov 03 23:06:04.995648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:04.995631 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.050757 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.050688 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:05.050757 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.050705 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:06:05.054069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.054044 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf"] Nov 03 23:06:05.182330 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.182295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.182796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.182368 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.182796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.182540 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfxx\" (UniqueName: \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.272284 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.272235 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" containerID="cri-o://dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20" gracePeriod=25 Nov 03 23:06:05.284183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.284151 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.284331 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.284218 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.284331 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.284287 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfxx\" (UniqueName: \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.284797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.284773 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.286660 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.286642 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.345509 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.345427 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfxx\" (UniqueName: \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.382728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.382691 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" containerID="cri-o://c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6" gracePeriod=25 Nov 03 23:06:05.609729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.609703 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:05.994373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:05.994345 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf"] Nov 03 23:06:05.996339 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:05.996311 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8c778c_baf8_486e_a9cd_b6cb4166d9a5.slice/crio-f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5 WatchSource:0}: Error finding container f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5: Status 404 returned error can't find the container with id f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5 Nov 03 23:06:06.411717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:06.411685 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0908daf7-a76e-463d-bfc7-7a7ea80806fa" path="/var/lib/kubelet/pods/0908daf7-a76e-463d-bfc7-7a7ea80806fa/volumes" Nov 03 23:06:06.785371 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:06.785331 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" event={"ID":"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5","Type":"ContainerStarted","Data":"dc9d3f47360f1345ced2c5d5c30f6152a3c37615ca553be9e47bfb73b20d42f7"} Nov 03 23:06:06.785584 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:06.785390 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" event={"ID":"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5","Type":"ContainerStarted","Data":"f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5"} Nov 03 23:06:06.901738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:06.901681 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" podStartSLOduration=2.901666504 podStartE2EDuration="2.901666504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:06.899852981 +0000 UTC m=+2640.637197207" watchObservedRunningTime="2026-11-03 23:06:06.901666504 +0000 UTC m=+2640.639010726" Nov 03 23:06:07.798097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:07.798054 5051 generic.go:356] "Generic (PLEG): container finished" podID="2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" containerID="dc9d3f47360f1345ced2c5d5c30f6152a3c37615ca553be9e47bfb73b20d42f7" exitCode=0 Nov 03 23:06:07.798532 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:07.798141 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" event={"ID":"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5","Type":"ContainerDied","Data":"dc9d3f47360f1345ced2c5d5c30f6152a3c37615ca553be9e47bfb73b20d42f7"} Nov 03 23:06:08.128957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.128917 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:08.129157 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.128918 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:08.262769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.262706 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" containerID="cri-o://bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117" gracePeriod=45 Nov 03 23:06:08.263445 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.263032 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" containerID="cri-o://942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb" gracePeriod=45 Nov 03 23:06:08.548626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.548595 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 23:06:08.567576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.567551 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 23:06:08.627560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627527 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.627754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627610 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\" (UniqueName: \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.627754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627670 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.627885 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627775 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.627885 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627815 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.627885 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627876 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.628111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627931 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.628111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.627982 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.628111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628069 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.628111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.628111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628097 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\") pod \"e434e52e-2636-4146-a822-ce3deee299bd\" (UID: \"e434e52e-2636-4146-a822-ce3deee299bd\") " Nov 03 23:06:08.628380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628187 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\" (UniqueName: \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.628380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628224 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.628380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628230 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.628380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628233 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\") pod \"d0103607-e3d9-4027-8398-657c568682ba\" (UID: \"d0103607-e3d9-4027-8398-657c568682ba\") " Nov 03 23:06:08.628380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628259 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.629131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.628848 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.629131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.629001 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.629131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.629015 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.629131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.629029 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.629417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.629263 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.629417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.629362 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.630845 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.630819 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "kube-api-access-ffv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:08.630845 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.630821 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.631037 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.630821 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:08.631576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.631444 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "kube-api-access-dx5qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:08.631576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.631468 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs" pod "d0103607-e3d9-4027-8398-657c568682ba" (UID: "d0103607-e3d9-4027-8398-657c568682ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:08.631576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.631498 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs" pod "e434e52e-2636-4146-a822-ce3deee299bd" (UID: "e434e52e-2636-4146-a822-ce3deee299bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:08.729858 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729825 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5qt\" (UniqueName: \"kubernetes.io/projected/d0103607-e3d9-4027-8398-657c568682ba-kube-api-access-dx5qt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.729858 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729857 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729868 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e434e52e-2636-4146-a822-ce3deee299bd-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729878 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffv9m\" (UniqueName: \"kubernetes.io/projected/e434e52e-2636-4146-a822-ce3deee299bd-kube-api-access-ffv9m\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729890 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729904 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0103607-e3d9-4027-8398-657c568682ba-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729912 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0103607-e3d9-4027-8398-657c568682ba-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.730109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.729919 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e434e52e-2636-4146-a822-ce3deee299bd-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:08.812153 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.812116 5051 generic.go:356] "Generic (PLEG): container finished" podID="e434e52e-2636-4146-a822-ce3deee299bd" containerID="bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117" exitCode=0 Nov 03 23:06:08.812653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.812199 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" Nov 03 23:06:08.812653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.812244 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" event={"ID":"e434e52e-2636-4146-a822-ce3deee299bd","Type":"ContainerDied","Data":"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117"} Nov 03 23:06:08.812653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.812280 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml" event={"ID":"e434e52e-2636-4146-a822-ce3deee299bd","Type":"ContainerDied","Data":"ad06d00d3eb97807414c4835515833a743a771c754daeb2dab8fa00a97af0769"} Nov 03 23:06:08.812653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.812303 5051 scope.go:122] "RemoveContainer" containerID="bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117" Nov 03 23:06:08.822307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.822275 5051 generic.go:356] "Generic (PLEG): container finished" podID="d0103607-e3d9-4027-8398-657c568682ba" containerID="942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb" exitCode=0 Nov 03 23:06:08.823729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.822321 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" Nov 03 23:06:08.823729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.822746 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" event={"ID":"d0103607-e3d9-4027-8398-657c568682ba","Type":"ContainerDied","Data":"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb"} Nov 03 23:06:08.823729 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.822787 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt" event={"ID":"d0103607-e3d9-4027-8398-657c568682ba","Type":"ContainerDied","Data":"99586abf97fc872a0b6068ba5cc439f86a5f19f8386b8e479d140d5447cfd73d"} Nov 03 23:06:08.828852 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.828758 5051 scope.go:122] "RemoveContainer" containerID="bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117" Nov 03 23:06:08.829188 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:08.829160 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117\": container with ID starting with bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117 not found: ID does not exist" containerID="bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117" Nov 03 23:06:08.829188 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.829196 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117"} err="failed to get container status \"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117\": rpc error: code = NotFound desc = could not find container \"bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117\": container with ID starting with bfb6c71b75c5f963d9d94bba89516a05ec3106e946de552fa7b8b59bb420d117 not found: ID does not exist" Nov 03 23:06:08.829188 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.829212 5051 scope.go:122] "RemoveContainer" containerID="942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb" Nov 03 23:06:08.848447 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.848405 5051 scope.go:122] "RemoveContainer" containerID="942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb" Nov 03 23:06:08.848852 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:08.848818 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb\": container with ID starting with 942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb not found: ID does not exist" containerID="942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb" Nov 03 23:06:08.848964 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.848864 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb"} err="failed to get container status \"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb\": rpc error: code = NotFound desc = could not find container \"942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb\": container with ID starting with 942f97a6c6042083bbffc2c91c8ee30476097304b227aa31f734e000c1e22dfb not found: ID does not exist" Nov 03 23:06:08.980090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.980061 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 23:06:08.986725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:08.986692 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-w6-7540da-kserve-84d988f488-9h9pt"] Nov 03 23:06:09.015129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.015070 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:09.036067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.036040 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\" (UniqueName: \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\") pod \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " Nov 03 23:06:09.036223 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.036135 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\") pod \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " Nov 03 23:06:09.036223 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.036186 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\") pod \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\" (UID: \"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5\") " Nov 03 23:06:09.036541 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.036515 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca" pod "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" (UID: "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:09.036763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.036747 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.038338 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.038320 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca" pod "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" (UID: "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:09.038403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.038390 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx" pod "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" (UID: "2f8c778c-baf8-486e-a9cd-b6cb4166d9a5"). InnerVolumeSpecName "kube-api-access-zwfxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:09.094134 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.094106 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 23:06:09.100056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.100018 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7540da/distinct-2-w6-7540da-kserve-757c67b5ff-z4sml"] Nov 03 23:06:09.138001 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.137968 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfxx\" (UniqueName: \"kubernetes.io/projected/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-kube-api-access-zwfxx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.138150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.138008 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.830565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.830528 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" Nov 03 23:06:09.830565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.830550 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf" event={"ID":"2f8c778c-baf8-486e-a9cd-b6cb4166d9a5","Type":"ContainerDied","Data":"f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5"} Nov 03 23:06:09.831175 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:09.830596 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f994efc41992c3f40ce7623c0222fcd192bc9aa9c3695ea75e1d519ca23b02f5" Nov 03 23:06:10.407808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:10.407774 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0103607-e3d9-4027-8398-657c568682ba" path="/var/lib/kubelet/pods/d0103607-e3d9-4027-8398-657c568682ba/volumes" Nov 03 23:06:10.408245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:10.408231 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e434e52e-2636-4146-a822-ce3deee299bd" path="/var/lib/kubelet/pods/e434e52e-2636-4146-a822-ce3deee299bd/volumes" Nov 03 23:06:11.240026 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:11.239984 5051 scope.go:122] "RemoveContainer" containerID="248d7e05846a4c5065f368d2e352bb1a8642e8872c9d54054398fe672bec1e03" Nov 03 23:06:16.543645 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:16.543553 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 23:06:16.648252 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:16.648214 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 23:06:18.129281 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:18.129239 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:18.129781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:18.129244 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:21.544576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.544541 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" podUID="206aad65-1e1b-455e-bb78-383be74653af" containerName="payload-processing" containerID="cri-o://505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7" gracePeriod=25 Nov 03 23:06:21.625110 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:21.625068 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:06:21.648900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.648854 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" podUID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" containerName="payload-pre-processing" containerID="cri-o://3080b3d241a272b43edf4e8d072f9a1fe87a6e90b18d355e783284b2cacfed5b" gracePeriod=25 Nov 03 23:06:21.867056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.867022 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 23:06:21.925151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.925113 5051 generic.go:356] "Generic (PLEG): container finished" podID="206aad65-1e1b-455e-bb78-383be74653af" containerID="505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7" exitCode=0 Nov 03 23:06:21.925376 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.925165 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" Nov 03 23:06:21.925376 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.925191 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" event={"ID":"206aad65-1e1b-455e-bb78-383be74653af","Type":"ContainerDied","Data":"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7"} Nov 03 23:06:21.925376 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.925247 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn" event={"ID":"206aad65-1e1b-455e-bb78-383be74653af","Type":"ContainerDied","Data":"4572d2d50f5dce92b394769f27f9d4476e36e2fa81cd1477d4ae84a5b3101620"} Nov 03 23:06:21.925376 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.925278 5051 scope.go:122] "RemoveContainer" containerID="505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7" Nov 03 23:06:21.927124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.927098 5051 generic.go:356] "Generic (PLEG): container finished" podID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" containerID="3080b3d241a272b43edf4e8d072f9a1fe87a6e90b18d355e783284b2cacfed5b" exitCode=0 Nov 03 23:06:21.927293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.927182 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" event={"ID":"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c","Type":"ContainerDied","Data":"3080b3d241a272b43edf4e8d072f9a1fe87a6e90b18d355e783284b2cacfed5b"} Nov 03 23:06:21.940133 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.940110 5051 scope.go:122] "RemoveContainer" containerID="505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7" Nov 03 23:06:21.940487 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:21.940464 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7\": container with ID starting with 505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7 not found: ID does not exist" containerID="505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7" Nov 03 23:06:21.940551 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.940496 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7"} err="failed to get container status \"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7\": rpc error: code = NotFound desc = could not find container \"505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7\": container with ID starting with 505d58a51debed9290429460bbe0288fb903b7778a859388acd5f4de4709a3f7 not found: ID does not exist" Nov 03 23:06:21.966483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.966461 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 23:06:21.987371 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.987343 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\") pod \"206aad65-1e1b-455e-bb78-383be74653af\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " Nov 03 23:06:21.987575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.987429 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\") pod \"206aad65-1e1b-455e-bb78-383be74653af\" (UID: \"206aad65-1e1b-455e-bb78-383be74653af\") " Nov 03 23:06:21.987825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.987798 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume" pod "206aad65-1e1b-455e-bb78-383be74653af" (UID: "206aad65-1e1b-455e-bb78-383be74653af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:21.988049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.988030 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206aad65-1e1b-455e-bb78-383be74653af-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:21.989862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:21.989834 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp" pod "206aad65-1e1b-455e-bb78-383be74653af" (UID: "206aad65-1e1b-455e-bb78-383be74653af"). InnerVolumeSpecName "kube-api-access-8dscp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:22.088708 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.088621 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\") pod \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " Nov 03 23:06:22.088902 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.088746 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\" (UniqueName: \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\") pod \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\" (UID: \"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c\") " Nov 03 23:06:22.089095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.089071 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume" pod "e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" (UID: "e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:22.089332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.089315 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:22.089395 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.089341 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/206aad65-1e1b-455e-bb78-383be74653af-kube-api-access-8dscp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:22.091051 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.091030 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx" pod "e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" (UID: "e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c"). InnerVolumeSpecName "kube-api-access-vj6lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:22.191187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.191153 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj6lx\" (UniqueName: \"kubernetes.io/projected/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c-kube-api-access-vj6lx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:22.351078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.347437 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 23:06:22.355516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.355480 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c93e70-5ffc55b946-vqsqn"] Nov 03 23:06:22.407417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.407384 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206aad65-1e1b-455e-bb78-383be74653af" path="/var/lib/kubelet/pods/206aad65-1e1b-455e-bb78-383be74653af/volumes" Nov 03 23:06:22.936701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.936675 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" Nov 03 23:06:22.936701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.936695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98" event={"ID":"e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c","Type":"ContainerDied","Data":"3baf19adf109bd367217e167fa63fdeda380bc64584b63b866437a72798f7c56"} Nov 03 23:06:22.937254 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:22.936753 5051 scope.go:122] "RemoveContainer" containerID="3080b3d241a272b43edf4e8d072f9a1fe87a6e90b18d355e783284b2cacfed5b" Nov 03 23:06:23.101901 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:23.101868 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 23:06:23.151445 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:23.151411 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c93e70-8fbcf64c-bvn98"] Nov 03 23:06:24.407617 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:24.407581 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" path="/var/lib/kubelet/pods/e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c/volumes" Nov 03 23:06:25.569020 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.568961 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 23:06:25.633918 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.633889 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\") pod \"e541be41-d501-4287-8a8d-6d5695aaf730\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " Nov 03 23:06:25.634109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.633948 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\" (UniqueName: \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\") pod \"e541be41-d501-4287-8a8d-6d5695aaf730\" (UID: \"e541be41-d501-4287-8a8d-6d5695aaf730\") " Nov 03 23:06:25.634326 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.634292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume" pod "e541be41-d501-4287-8a8d-6d5695aaf730" (UID: "e541be41-d501-4287-8a8d-6d5695aaf730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:25.634437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.634417 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e541be41-d501-4287-8a8d-6d5695aaf730-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:25.636389 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.636360 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk" pod "e541be41-d501-4287-8a8d-6d5695aaf730" (UID: "e541be41-d501-4287-8a8d-6d5695aaf730"). InnerVolumeSpecName "kube-api-access-g8tpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:25.651493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.651472 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 23:06:25.735663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.735630 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\") pod \"84397155-43d0-4d44-b3f2-285f6c39ba81\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " Nov 03 23:06:25.735857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.735802 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\" (UniqueName: \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\") pod \"84397155-43d0-4d44-b3f2-285f6c39ba81\" (UID: \"84397155-43d0-4d44-b3f2-285f6c39ba81\") " Nov 03 23:06:25.736133 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.736106 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume" pod "84397155-43d0-4d44-b3f2-285f6c39ba81" (UID: "84397155-43d0-4d44-b3f2-285f6c39ba81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:25.736408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.736391 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tpk\" (UniqueName: \"kubernetes.io/projected/e541be41-d501-4287-8a8d-6d5695aaf730-kube-api-access-g8tpk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:25.736484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.736410 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84397155-43d0-4d44-b3f2-285f6c39ba81-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:25.738134 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.738113 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d" pod "84397155-43d0-4d44-b3f2-285f6c39ba81" (UID: "84397155-43d0-4d44-b3f2-285f6c39ba81"). InnerVolumeSpecName "kube-api-access-f4c7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:25.837174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.837094 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c7d\" (UniqueName: \"kubernetes.io/projected/84397155-43d0-4d44-b3f2-285f6c39ba81-kube-api-access-f4c7d\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:25.959436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.959403 5051 generic.go:356] "Generic (PLEG): container finished" podID="e541be41-d501-4287-8a8d-6d5695aaf730" containerID="dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20" exitCode=0 Nov 03 23:06:25.959603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.959447 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" Nov 03 23:06:25.959603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.959490 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" event={"ID":"e541be41-d501-4287-8a8d-6d5695aaf730","Type":"ContainerDied","Data":"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20"} Nov 03 23:06:25.959603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.959545 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt" event={"ID":"e541be41-d501-4287-8a8d-6d5695aaf730","Type":"ContainerDied","Data":"edb80754a69888fe7f801ce6544b3dc442db4bd203192ea7038556fdf1503267"} Nov 03 23:06:25.959603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.959573 5051 scope.go:122] "RemoveContainer" containerID="dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20" Nov 03 23:06:25.961251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.961196 5051 generic.go:356] "Generic (PLEG): container finished" podID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerID="c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6" exitCode=0 Nov 03 23:06:25.961251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.961229 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" event={"ID":"84397155-43d0-4d44-b3f2-285f6c39ba81","Type":"ContainerDied","Data":"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6"} Nov 03 23:06:25.961396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.961280 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" event={"ID":"84397155-43d0-4d44-b3f2-285f6c39ba81","Type":"ContainerDied","Data":"16db0057280f00ef6c026c6ce00d953993c74daf686f0ecb098ccb82bca52e45"} Nov 03 23:06:25.961396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.961238 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm" Nov 03 23:06:25.972452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.972427 5051 scope.go:122] "RemoveContainer" containerID="dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20" Nov 03 23:06:25.972816 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:25.972787 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20\": container with ID starting with dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20 not found: ID does not exist" containerID="dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20" Nov 03 23:06:25.972933 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.972827 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20"} err="failed to get container status \"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20\": rpc error: code = NotFound desc = could not find container \"dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20\": container with ID starting with dee9a73ad2e341269daa5de2353de414f40670129debfa1867bae71b5203ee20 not found: ID does not exist" Nov 03 23:06:25.972933 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.972852 5051 scope.go:122] "RemoveContainer" containerID="c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6" Nov 03 23:06:25.984195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.984167 5051 scope.go:122] "RemoveContainer" containerID="c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6" Nov 03 23:06:25.984563 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:25.984531 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6\": container with ID starting with c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6 not found: ID does not exist" containerID="c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6" Nov 03 23:06:25.984666 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:25.984575 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6"} err="failed to get container status \"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6\": rpc error: code = NotFound desc = could not find container \"c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6\": container with ID starting with c4223c058fc0771a933a714146f8e81763af703d4669636069ce69e0057810f6 not found: ID does not exist" Nov 03 23:06:26.130818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.130743 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 23:06:26.135034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.134982 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7540da-59844f9b7b-z68qt"] Nov 03 23:06:26.244018 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.243829 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 23:06:26.273968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.267554 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7540da-598d566c67-ld8zm"] Nov 03 23:06:26.412732 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.412641 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" path="/var/lib/kubelet/pods/84397155-43d0-4d44-b3f2-285f6c39ba81/volumes" Nov 03 23:06:26.413277 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:26.413257 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" path="/var/lib/kubelet/pods/e541be41-d501-4287-8a8d-6d5695aaf730/volumes" Nov 03 23:06:27.972945 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:27.972908 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n"] Nov 03 23:06:27.982938 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:27.982901 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-78be2c-llq7n"] Nov 03 23:06:28.408822 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:28.408785 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366e37ac-40f9-4029-8c65-12cdcad1c444" path="/var/lib/kubelet/pods/366e37ac-40f9-4029-8c65-12cdcad1c444/volumes" Nov 03 23:06:36.170611 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.170577 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4"] Nov 03 23:06:36.171120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171107 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171121 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171138 5051 cpu_manager.go:414] "containerMap: removing container" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171143 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171153 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" containerName="revoke-keys" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171158 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" containerName="revoke-keys" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171165 5051 cpu_manager.go:414] "containerMap: removing container" podUID="206aad65-1e1b-455e-bb78-383be74653af" containerName="payload-processing" Nov 03 23:06:36.171169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171170 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206aad65-1e1b-455e-bb78-383be74653af" containerName="payload-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171195 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171200 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171208 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" containerName="payload-pre-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171212 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" containerName="payload-pre-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171225 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171230 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171313 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0103607-e3d9-4027-8398-657c568682ba" containerName="main" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171322 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24fb1fc-c5c1-4e0d-8b65-408f2e9e5b7c" containerName="payload-pre-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171330 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e541be41-d501-4287-8a8d-6d5695aaf730" containerName="payload-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171337 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e434e52e-2636-4146-a822-ce3deee299bd" containerName="main" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171342 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="84397155-43d0-4d44-b3f2-285f6c39ba81" containerName="payload-pre-processing" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171348 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" containerName="revoke-keys" Nov 03 23:06:36.171388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.171354 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="206aad65-1e1b-455e-bb78-383be74653af" containerName="payload-processing" Nov 03 23:06:36.174706 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.174688 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.232021 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.231967 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4"] Nov 03 23:06:36.276332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.276297 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:36.276332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.276332 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:06:36.291834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.291804 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzhm\" (UniqueName: \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.292007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.291920 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.292007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.291957 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.393444 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.393393 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.393444 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.393445 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.393731 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.393603 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzhm\" (UniqueName: \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.394389 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.394366 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.396566 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.396543 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.454416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.454342 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzhm\" (UniqueName: \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.488045 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.487981 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:36.870872 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:36.870847 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4"] Nov 03 23:06:36.872691 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:36.872659 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1884eeb_a781_4439_8fc1_6c1291f6a39e.slice/crio-c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c WatchSource:0}: Error finding container c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c: Status 404 returned error can't find the container with id c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c Nov 03 23:06:37.051755 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:37.051721 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" event={"ID":"c1884eeb-a781-4439-8fc1-6c1291f6a39e","Type":"ContainerStarted","Data":"c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c"} Nov 03 23:06:38.062474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:38.062435 5051 generic.go:356] "Generic (PLEG): container finished" podID="c1884eeb-a781-4439-8fc1-6c1291f6a39e" containerID="d372d97e74a03521ae56edd18d9946f4315e77e5194c5a81eeb6a101047ff3a3" exitCode=0 Nov 03 23:06:38.062898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:38.062523 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" event={"ID":"c1884eeb-a781-4439-8fc1-6c1291f6a39e","Type":"ContainerDied","Data":"d372d97e74a03521ae56edd18d9946f4315e77e5194c5a81eeb6a101047ff3a3"} Nov 03 23:06:39.232827 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.232795 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:39.426686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.426656 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\") pod \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " Nov 03 23:06:39.426910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.426801 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\" (UniqueName: \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\") pod \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " Nov 03 23:06:39.426910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.426838 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\") pod \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\" (UID: \"c1884eeb-a781-4439-8fc1-6c1291f6a39e\") " Nov 03 23:06:39.427130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.427107 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca" pod "c1884eeb-a781-4439-8fc1-6c1291f6a39e" (UID: "c1884eeb-a781-4439-8fc1-6c1291f6a39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:39.427453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.427418 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:39.429336 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.429309 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca" pod "c1884eeb-a781-4439-8fc1-6c1291f6a39e" (UID: "c1884eeb-a781-4439-8fc1-6c1291f6a39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:39.429510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.429490 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm" pod "c1884eeb-a781-4439-8fc1-6c1291f6a39e" (UID: "c1884eeb-a781-4439-8fc1-6c1291f6a39e"). InnerVolumeSpecName "kube-api-access-4mzhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:39.529080 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.529037 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mzhm\" (UniqueName: \"kubernetes.io/projected/c1884eeb-a781-4439-8fc1-6c1291f6a39e-kube-api-access-4mzhm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:39.529080 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:39.529076 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1884eeb-a781-4439-8fc1-6c1291f6a39e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:40.078423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:40.078392 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" Nov 03 23:06:40.078630 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:40.078387 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4" event={"ID":"c1884eeb-a781-4439-8fc1-6c1291f6a39e","Type":"ContainerDied","Data":"c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c"} Nov 03 23:06:40.078630 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:40.078530 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e70a2e4dc8d8dc44df3381275525f134cd05a6701a3239bcadc572caea283c" Nov 03 23:06:42.526181 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.526143 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr"] Nov 03 23:06:42.527116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.527085 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c1884eeb-a781-4439-8fc1-6c1291f6a39e" containerName="revoke-keys" Nov 03 23:06:42.527116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.527114 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1884eeb-a781-4439-8fc1-6c1291f6a39e" containerName="revoke-keys" Nov 03 23:06:42.527320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.527312 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1884eeb-a781-4439-8fc1-6c1291f6a39e" containerName="revoke-keys" Nov 03 23:06:42.530815 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.530798 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.557601 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.557571 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.557797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.557613 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9sgl\" (UniqueName: \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.557797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.557764 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.583774 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.583740 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj"] Nov 03 23:06:42.588262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.588232 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.589152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.589124 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr"] Nov 03 23:06:42.626980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.626950 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:42.627146 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.627047 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:06:42.643900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.643868 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj"] Nov 03 23:06:42.659066 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659026 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9sgl\" (UniqueName: \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.659245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659097 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.659245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659199 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.659385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659345 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.659385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659371 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spstz\" (UniqueName: \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.659534 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.659510 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.660513 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.660484 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.662131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.662103 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.720049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.720021 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9sgl\" (UniqueName: \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\") pod \"maas-api-revoke-sub-c3350a37-ktlxr\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.760539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.760502 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.760539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.760539 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spstz\" (UniqueName: \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.760780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.760641 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.761355 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.761334 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.763224 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.763196 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.821615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.821540 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spstz\" (UniqueName: \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\") pod \"maas-api-revoke-sub-87fee90a-s82xj\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:42.848943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.845403 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:42.912647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:42.912614 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:43.254531 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:43.254492 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr"] Nov 03 23:06:43.307171 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:43.307129 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39086f00_2822_4359_8c85_bb6d6b7bca3e.slice/crio-ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0 WatchSource:0}: Error finding container ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0: Status 404 returned error can't find the container with id ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0 Nov 03 23:06:43.367678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:43.367651 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj"] Nov 03 23:06:43.371971 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:43.371947 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc379af8_b66b_4daf_90ad_4d34fb00dca3.slice/crio-a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a WatchSource:0}: Error finding container a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a: Status 404 returned error can't find the container with id a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a Nov 03 23:06:44.115553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.115514 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" event={"ID":"39086f00-2822-4359-8c85-bb6d6b7bca3e","Type":"ContainerStarted","Data":"d535afce83aca3fe7dea523e617938f2e40e1916c06fe4084c526fa6c27bc86f"} Nov 03 23:06:44.116120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.115577 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" event={"ID":"39086f00-2822-4359-8c85-bb6d6b7bca3e","Type":"ContainerStarted","Data":"ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0"} Nov 03 23:06:44.117581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.117558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" event={"ID":"bc379af8-b66b-4daf-90ad-4d34fb00dca3","Type":"ContainerStarted","Data":"05cc4acff94f20e8dceac49041010890ce506aad811e2b8310ca45ba7a3642a1"} Nov 03 23:06:44.117693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.117594 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" event={"ID":"bc379af8-b66b-4daf-90ad-4d34fb00dca3","Type":"ContainerStarted","Data":"a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a"} Nov 03 23:06:44.231943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.231887 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" podStartSLOduration=2.231872685 podStartE2EDuration="2.231872685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:44.229959301 +0000 UTC m=+2677.967303526" watchObservedRunningTime="2026-11-03 23:06:44.231872685 +0000 UTC m=+2677.969216932" Nov 03 23:06:44.348341 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.348278 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" podStartSLOduration=2.348263155 podStartE2EDuration="2.348263155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:44.344641957 +0000 UTC m=+2678.081986181" watchObservedRunningTime="2026-11-03 23:06:44.348263155 +0000 UTC m=+2678.085607382" Nov 03 23:06:44.940738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.940706 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf"] Nov 03 23:06:44.989527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:44.989493 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c93e70-t7wdf"] Nov 03 23:06:45.125228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:45.125193 5051 generic.go:356] "Generic (PLEG): container finished" podID="bc379af8-b66b-4daf-90ad-4d34fb00dca3" containerID="05cc4acff94f20e8dceac49041010890ce506aad811e2b8310ca45ba7a3642a1" exitCode=0 Nov 03 23:06:45.125687 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:45.125279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" event={"ID":"bc379af8-b66b-4daf-90ad-4d34fb00dca3","Type":"ContainerDied","Data":"05cc4acff94f20e8dceac49041010890ce506aad811e2b8310ca45ba7a3642a1"} Nov 03 23:06:45.126905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:45.126886 5051 generic.go:356] "Generic (PLEG): container finished" podID="39086f00-2822-4359-8c85-bb6d6b7bca3e" containerID="d535afce83aca3fe7dea523e617938f2e40e1916c06fe4084c526fa6c27bc86f" exitCode=0 Nov 03 23:06:45.127013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:45.126915 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" event={"ID":"39086f00-2822-4359-8c85-bb6d6b7bca3e","Type":"ContainerDied","Data":"d535afce83aca3fe7dea523e617938f2e40e1916c06fe4084c526fa6c27bc86f"} Nov 03 23:06:46.292182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.292161 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:46.308552 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.308528 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\") pod \"39086f00-2822-4359-8c85-bb6d6b7bca3e\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " Nov 03 23:06:46.308708 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.308692 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\" (UniqueName: \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\") pod \"39086f00-2822-4359-8c85-bb6d6b7bca3e\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " Nov 03 23:06:46.308773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.308745 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\") pod \"39086f00-2822-4359-8c85-bb6d6b7bca3e\" (UID: \"39086f00-2822-4359-8c85-bb6d6b7bca3e\") " Nov 03 23:06:46.308939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.308915 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca" pod "39086f00-2822-4359-8c85-bb6d6b7bca3e" (UID: "39086f00-2822-4359-8c85-bb6d6b7bca3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:46.309262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.309243 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.311132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.311109 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca" pod "39086f00-2822-4359-8c85-bb6d6b7bca3e" (UID: "39086f00-2822-4359-8c85-bb6d6b7bca3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:46.311345 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.311321 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl" pod "39086f00-2822-4359-8c85-bb6d6b7bca3e" (UID: "39086f00-2822-4359-8c85-bb6d6b7bca3e"). InnerVolumeSpecName "kube-api-access-l9sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:46.326170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.326149 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:46.408665 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.408591 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8c778c-baf8-486e-a9cd-b6cb4166d9a5" path="/var/lib/kubelet/pods/2f8c778c-baf8-486e-a9cd-b6cb4166d9a5/volumes" Nov 03 23:06:46.409597 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.409577 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\") pod \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " Nov 03 23:06:46.409747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.409647 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\") pod \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " Nov 03 23:06:46.409823 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.409771 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\" (UniqueName: \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\") pod \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\" (UID: \"bc379af8-b66b-4daf-90ad-4d34fb00dca3\") " Nov 03 23:06:46.409982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.409957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca" pod "bc379af8-b66b-4daf-90ad-4d34fb00dca3" (UID: "bc379af8-b66b-4daf-90ad-4d34fb00dca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:46.410308 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.410287 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9sgl\" (UniqueName: \"kubernetes.io/projected/39086f00-2822-4359-8c85-bb6d6b7bca3e-kube-api-access-l9sgl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.410308 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.410308 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39086f00-2822-4359-8c85-bb6d6b7bca3e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.410486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.410318 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.414153 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.414130 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca" pod "bc379af8-b66b-4daf-90ad-4d34fb00dca3" (UID: "bc379af8-b66b-4daf-90ad-4d34fb00dca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:46.414276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.414157 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz" pod "bc379af8-b66b-4daf-90ad-4d34fb00dca3" (UID: "bc379af8-b66b-4daf-90ad-4d34fb00dca3"). InnerVolumeSpecName "kube-api-access-spstz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:46.511251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.511222 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spstz\" (UniqueName: \"kubernetes.io/projected/bc379af8-b66b-4daf-90ad-4d34fb00dca3-kube-api-access-spstz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:46.511251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:46.511249 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc379af8-b66b-4daf-90ad-4d34fb00dca3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:47.142178 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.142139 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" Nov 03 23:06:47.142365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.142176 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj" event={"ID":"bc379af8-b66b-4daf-90ad-4d34fb00dca3","Type":"ContainerDied","Data":"a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a"} Nov 03 23:06:47.142365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.142225 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4987d1b913117c69f535eff70764e4127c356e849b039515a05967b40a3c59a" Nov 03 23:06:47.143661 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.143633 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" Nov 03 23:06:47.143803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.143678 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr" event={"ID":"39086f00-2822-4359-8c85-bb6d6b7bca3e","Type":"ContainerDied","Data":"ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0"} Nov 03 23:06:47.143803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.143716 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef062be08ddf7f646a9572763d81a66386f3c17d006f434b14401e3a8ca7afa0" Nov 03 23:06:47.666528 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.666494 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 23:06:47.774253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:47.774214 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 23:06:51.366034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:51.365978 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9"] Nov 03 23:06:51.415058 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:51.415027 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1048fc-bppm9"] Nov 03 23:06:51.624196 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:51.624107 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:06:52.410248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.410212 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5c3b12-180c-4364-817b-9a72aaf06699" path="/var/lib/kubelet/pods/0a5c3b12-180c-4364-817b-9a72aaf06699/volumes" Nov 03 23:06:52.667432 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.667340 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" podUID="42a148d1-c86d-49c7-b93b-95ec72de33a1" containerName="payload-processing" containerID="cri-o://d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4" gracePeriod=25 Nov 03 23:06:52.774817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.774768 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" podUID="b933528a-d3df-4255-940d-6fc634b58699" containerName="payload-pre-processing" containerID="cri-o://ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e" gracePeriod=25 Nov 03 23:06:52.800639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.800607 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx"] Nov 03 23:06:52.801435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801415 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bc379af8-b66b-4daf-90ad-4d34fb00dca3" containerName="revoke-keys" Nov 03 23:06:52.801519 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801439 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc379af8-b66b-4daf-90ad-4d34fb00dca3" containerName="revoke-keys" Nov 03 23:06:52.801519 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801477 5051 cpu_manager.go:414] "containerMap: removing container" podUID="39086f00-2822-4359-8c85-bb6d6b7bca3e" containerName="revoke-keys" Nov 03 23:06:52.801519 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801486 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39086f00-2822-4359-8c85-bb6d6b7bca3e" containerName="revoke-keys" Nov 03 23:06:52.801641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801619 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc379af8-b66b-4daf-90ad-4d34fb00dca3" containerName="revoke-keys" Nov 03 23:06:52.801641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.801635 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="39086f00-2822-4359-8c85-bb6d6b7bca3e" containerName="revoke-keys" Nov 03 23:06:52.805327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.805309 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:52.916070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.908838 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx"] Nov 03 23:06:52.916070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.913354 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:06:52.916070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.913426 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:52.940092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.939272 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k72ft\" (UniqueName: \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:52.940092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.939427 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:52.940092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:52.939550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.011350 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.011321 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 23:06:53.040868 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.040837 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.041095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.040952 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.041095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.041025 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k72ft\" (UniqueName: \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.042085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.042054 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.043890 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.043864 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.102103 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.102072 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 23:06:53.103081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.103060 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k72ft\" (UniqueName: \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.141556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.141531 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\" (UniqueName: \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\") pod \"42a148d1-c86d-49c7-b93b-95ec72de33a1\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " Nov 03 23:06:53.141689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.141649 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\") pod \"b933528a-d3df-4255-940d-6fc634b58699\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " Nov 03 23:06:53.141788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.141774 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\" (UniqueName: \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\") pod \"b933528a-d3df-4255-940d-6fc634b58699\" (UID: \"b933528a-d3df-4255-940d-6fc634b58699\") " Nov 03 23:06:53.141924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.141835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\") pod \"42a148d1-c86d-49c7-b93b-95ec72de33a1\" (UID: \"42a148d1-c86d-49c7-b93b-95ec72de33a1\") " Nov 03 23:06:53.142030 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.141977 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume" pod "b933528a-d3df-4255-940d-6fc634b58699" (UID: "b933528a-d3df-4255-940d-6fc634b58699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:53.142274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.142254 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume" pod "42a148d1-c86d-49c7-b93b-95ec72de33a1" (UID: "42a148d1-c86d-49c7-b93b-95ec72de33a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:53.142392 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.142378 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b933528a-d3df-4255-940d-6fc634b58699-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.142440 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.142394 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a148d1-c86d-49c7-b93b-95ec72de33a1-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.143738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.143716 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw" pod "42a148d1-c86d-49c7-b93b-95ec72de33a1" (UID: "42a148d1-c86d-49c7-b93b-95ec72de33a1"). InnerVolumeSpecName "kube-api-access-dsmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:53.143932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.143913 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6" pod "b933528a-d3df-4255-940d-6fc634b58699" (UID: "b933528a-d3df-4255-940d-6fc634b58699"). InnerVolumeSpecName "kube-api-access-zblc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:53.218094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.217984 5051 generic.go:356] "Generic (PLEG): container finished" podID="b933528a-d3df-4255-940d-6fc634b58699" containerID="ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e" exitCode=0 Nov 03 23:06:53.218094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.218046 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" Nov 03 23:06:53.218333 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.218100 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" event={"ID":"b933528a-d3df-4255-940d-6fc634b58699","Type":"ContainerDied","Data":"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e"} Nov 03 23:06:53.218333 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.218162 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz" event={"ID":"b933528a-d3df-4255-940d-6fc634b58699","Type":"ContainerDied","Data":"af1b6e349bb0d4152485ba934be05cc6f4faea37e43adee8bec22ae4a3acb252"} Nov 03 23:06:53.218333 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.218192 5051 scope.go:122] "RemoveContainer" containerID="ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e" Nov 03 23:06:53.219800 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.219778 5051 generic.go:356] "Generic (PLEG): container finished" podID="42a148d1-c86d-49c7-b93b-95ec72de33a1" containerID="d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4" exitCode=0 Nov 03 23:06:53.219942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.219821 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" Nov 03 23:06:53.219942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.219848 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" event={"ID":"42a148d1-c86d-49c7-b93b-95ec72de33a1","Type":"ContainerDied","Data":"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4"} Nov 03 23:06:53.219942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.219887 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64" event={"ID":"42a148d1-c86d-49c7-b93b-95ec72de33a1","Type":"ContainerDied","Data":"4e9c0a3f059d7a996d8864ad7c1fc3794e35e25f4e1d4ce566a432111993c9b1"} Nov 03 23:06:53.236566 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.236531 5051 scope.go:122] "RemoveContainer" containerID="ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e" Nov 03 23:06:53.236972 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:53.236942 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e\": container with ID starting with ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e not found: ID does not exist" containerID="ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e" Nov 03 23:06:53.237085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.237032 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e"} err="failed to get container status \"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e\": rpc error: code = NotFound desc = could not find container \"ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e\": container with ID starting with ccec3e4e280eef1191c912c900e7c806f4ce7f582cbe623097d45c690da8d31e not found: ID does not exist" Nov 03 23:06:53.237085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.237064 5051 scope.go:122] "RemoveContainer" containerID="d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4" Nov 03 23:06:53.241296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.241274 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:53.243904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.243884 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmvw\" (UniqueName: \"kubernetes.io/projected/42a148d1-c86d-49c7-b93b-95ec72de33a1-kube-api-access-dsmvw\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.243984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.243906 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblc6\" (UniqueName: \"kubernetes.io/projected/b933528a-d3df-4255-940d-6fc634b58699-kube-api-access-zblc6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:53.247412 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.247395 5051 scope.go:122] "RemoveContainer" containerID="d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4" Nov 03 23:06:53.247696 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:06:53.247677 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4\": container with ID starting with d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4 not found: ID does not exist" containerID="d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4" Nov 03 23:06:53.247745 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.247704 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4"} err="failed to get container status \"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4\": rpc error: code = NotFound desc = could not find container \"d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4\": container with ID starting with d10327b992e20ac957a1bccb38db6aa5e38d699f81354b30df646157244cf1e4 not found: ID does not exist" Nov 03 23:06:53.390486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.390457 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 23:06:53.393727 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.393696 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-eb3c33-6ff459c8c9-jfqlz"] Nov 03 23:06:53.501483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.501337 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 23:06:53.507070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.507038 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eb3c33-846894ccd8-mzt64"] Nov 03 23:06:53.630414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:53.630389 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx"] Nov 03 23:06:53.632542 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:53.632513 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12ed289_7bcc_4f4f_a0a8_3a36bea59852.slice/crio-3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2 WatchSource:0}: Error finding container 3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2: Status 404 returned error can't find the container with id 3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2 Nov 03 23:06:54.227973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:54.227880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" event={"ID":"d12ed289-7bcc-4f4f-a0a8-3a36bea59852","Type":"ContainerStarted","Data":"f7c442def84eee48fecbe68091fcb3531ee14e41004dd8f87e7f7657c20693a0"} Nov 03 23:06:54.227973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:54.227936 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" event={"ID":"d12ed289-7bcc-4f4f-a0a8-3a36bea59852","Type":"ContainerStarted","Data":"3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2"} Nov 03 23:06:54.346189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:54.346135 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" podStartSLOduration=2.34612093 podStartE2EDuration="2.34612093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:54.342691671 +0000 UTC m=+2688.080035897" watchObservedRunningTime="2026-11-03 23:06:54.34612093 +0000 UTC m=+2688.083465154" Nov 03 23:06:54.408386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:54.408353 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a148d1-c86d-49c7-b93b-95ec72de33a1" path="/var/lib/kubelet/pods/42a148d1-c86d-49c7-b93b-95ec72de33a1/volumes" Nov 03 23:06:54.408764 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:54.408751 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b933528a-d3df-4255-940d-6fc634b58699" path="/var/lib/kubelet/pods/b933528a-d3df-4255-940d-6fc634b58699/volumes" Nov 03 23:06:55.239343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:55.239308 5051 generic.go:356] "Generic (PLEG): container finished" podID="d12ed289-7bcc-4f4f-a0a8-3a36bea59852" containerID="f7c442def84eee48fecbe68091fcb3531ee14e41004dd8f87e7f7657c20693a0" exitCode=0 Nov 03 23:06:55.239739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:55.239394 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" event={"ID":"d12ed289-7bcc-4f4f-a0a8-3a36bea59852","Type":"ContainerDied","Data":"f7c442def84eee48fecbe68091fcb3531ee14e41004dd8f87e7f7657c20693a0"} Nov 03 23:06:56.388241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.388214 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:56.481865 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.481833 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\") pod \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " Nov 03 23:06:56.482078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.481915 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\") pod \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " Nov 03 23:06:56.482078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.482030 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\" (UniqueName: \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\") pod \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\" (UID: \"d12ed289-7bcc-4f4f-a0a8-3a36bea59852\") " Nov 03 23:06:56.482298 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.482276 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca" pod "d12ed289-7bcc-4f4f-a0a8-3a36bea59852" (UID: "d12ed289-7bcc-4f4f-a0a8-3a36bea59852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:56.482541 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.482525 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:56.484262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.484237 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca" pod "d12ed289-7bcc-4f4f-a0a8-3a36bea59852" (UID: "d12ed289-7bcc-4f4f-a0a8-3a36bea59852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:56.484367 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.484320 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft" pod "d12ed289-7bcc-4f4f-a0a8-3a36bea59852" (UID: "d12ed289-7bcc-4f4f-a0a8-3a36bea59852"). InnerVolumeSpecName "kube-api-access-k72ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:56.584169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.584055 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:56.584169 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:56.584085 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k72ft\" (UniqueName: \"kubernetes.io/projected/d12ed289-7bcc-4f4f-a0a8-3a36bea59852-kube-api-access-k72ft\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:57.255246 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:57.255208 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" event={"ID":"d12ed289-7bcc-4f4f-a0a8-3a36bea59852","Type":"ContainerDied","Data":"3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2"} Nov 03 23:06:57.255407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:57.255264 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7c7c148ba6ba3106a072e88cc9a195d1b6b01a3c6ca8f07de00ec007ff20e2" Nov 03 23:06:57.255407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:57.255228 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx" Nov 03 23:06:58.968526 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.968483 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf"] Nov 03 23:06:58.969071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969027 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d12ed289-7bcc-4f4f-a0a8-3a36bea59852" containerName="revoke-keys" Nov 03 23:06:58.969071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969040 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12ed289-7bcc-4f4f-a0a8-3a36bea59852" containerName="revoke-keys" Nov 03 23:06:58.969071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969060 5051 cpu_manager.go:414] "containerMap: removing container" podUID="42a148d1-c86d-49c7-b93b-95ec72de33a1" containerName="payload-processing" Nov 03 23:06:58.969071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969065 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a148d1-c86d-49c7-b93b-95ec72de33a1" containerName="payload-processing" Nov 03 23:06:58.969071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969073 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b933528a-d3df-4255-940d-6fc634b58699" containerName="payload-pre-processing" Nov 03 23:06:58.969400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969079 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b933528a-d3df-4255-940d-6fc634b58699" containerName="payload-pre-processing" Nov 03 23:06:58.969400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969163 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12ed289-7bcc-4f4f-a0a8-3a36bea59852" containerName="revoke-keys" Nov 03 23:06:58.969400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969173 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a148d1-c86d-49c7-b93b-95ec72de33a1" containerName="payload-processing" Nov 03 23:06:58.969400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.969179 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b933528a-d3df-4255-940d-6fc634b58699" containerName="payload-pre-processing" Nov 03 23:06:58.972894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:58.972873 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.033255 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.033211 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf"] Nov 03 23:06:59.075675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.075645 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:59.075856 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.075737 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:06:59.115314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.115279 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.115466 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.115339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.115524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.115463 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqtb\" (UniqueName: \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.216623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.216584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqtb\" (UniqueName: \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.216810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.216747 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.216810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.216798 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.217433 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.217409 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.219488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.219431 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.277668 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.277642 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqtb\" (UniqueName: \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\") pod \"maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.291775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.291746 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:06:59.672731 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:06:59.672697 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf"] Nov 03 23:06:59.674084 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:06:59.674045 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82809752_7420_4fd5_b86d_74e7c1ae7432.slice/crio-1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18 WatchSource:0}: Error finding container 1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18: Status 404 returned error can't find the container with id 1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18 Nov 03 23:07:00.289967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:00.289866 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" event={"ID":"82809752-7420-4fd5-b86d-74e7c1ae7432","Type":"ContainerStarted","Data":"e2f9236e154999e03a4ad77409ee6c90b5b9d132ccb736ec69a9746adf4ac778"} Nov 03 23:07:00.289967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:00.289926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" event={"ID":"82809752-7420-4fd5-b86d-74e7c1ae7432","Type":"ContainerStarted","Data":"1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18"} Nov 03 23:07:00.407173 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:00.407126 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" podStartSLOduration=2.407112084 podStartE2EDuration="2.407112084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:00.405395024 +0000 UTC m=+2694.142739245" watchObservedRunningTime="2026-11-03 23:07:00.407112084 +0000 UTC m=+2694.144456309" Nov 03 23:07:01.301488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:01.301439 5051 generic.go:356] "Generic (PLEG): container finished" podID="82809752-7420-4fd5-b86d-74e7c1ae7432" containerID="e2f9236e154999e03a4ad77409ee6c90b5b9d132ccb736ec69a9746adf4ac778" exitCode=0 Nov 03 23:07:01.301967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:01.301490 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" event={"ID":"82809752-7420-4fd5-b86d-74e7c1ae7432","Type":"ContainerDied","Data":"e2f9236e154999e03a4ad77409ee6c90b5b9d132ccb736ec69a9746adf4ac778"} Nov 03 23:07:02.467639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.467618 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:07:02.556325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.556283 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\") pod \"82809752-7420-4fd5-b86d-74e7c1ae7432\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " Nov 03 23:07:02.556535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.556415 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\") pod \"82809752-7420-4fd5-b86d-74e7c1ae7432\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " Nov 03 23:07:02.556535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.556462 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\" (UniqueName: \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\") pod \"82809752-7420-4fd5-b86d-74e7c1ae7432\" (UID: \"82809752-7420-4fd5-b86d-74e7c1ae7432\") " Nov 03 23:07:02.556691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.556660 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca" pod "82809752-7420-4fd5-b86d-74e7c1ae7432" (UID: "82809752-7420-4fd5-b86d-74e7c1ae7432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:02.556891 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.556868 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.558772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.558749 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca" pod "82809752-7420-4fd5-b86d-74e7c1ae7432" (UID: "82809752-7420-4fd5-b86d-74e7c1ae7432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:02.558876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.558856 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb" pod "82809752-7420-4fd5-b86d-74e7c1ae7432" (UID: "82809752-7420-4fd5-b86d-74e7c1ae7432"). InnerVolumeSpecName "kube-api-access-wxqtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:02.657984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.657948 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82809752-7420-4fd5-b86d-74e7c1ae7432-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:02.657984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:02.657983 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqtb\" (UniqueName: \"kubernetes.io/projected/82809752-7420-4fd5-b86d-74e7c1ae7432-kube-api-access-wxqtb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.324084 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:03.324057 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" Nov 03 23:07:03.324263 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:03.324054 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf" event={"ID":"82809752-7420-4fd5-b86d-74e7c1ae7432","Type":"ContainerDied","Data":"1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18"} Nov 03 23:07:03.324263 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:03.324213 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1926db8c6ded6a265d2894fd3467d87a3f45c66a45a0aeb67868006bdf700d18" Nov 03 23:07:07.272104 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:07.272080 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:07:07.303363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:07.303337 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:07:08.388158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:08.388125 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4"] Nov 03 23:07:08.435919 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:08.435885 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eb3c33-vn9n4"] Nov 03 23:07:10.410160 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:10.410126 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1884eeb-a781-4439-8fc1-6c1291f6a39e" path="/var/lib/kubelet/pods/c1884eeb-a781-4439-8fc1-6c1291f6a39e/volumes" Nov 03 23:07:15.244088 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:15.244046 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 23:07:15.244578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:15.244306 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" containerID="cri-o://ffcccc4c75de48c81b7d33c5f4daedb12c89293b5b52f4247adefdf3707d6ec4" gracePeriod=30 Nov 03 23:07:16.059437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.059403 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt"] Nov 03 23:07:16.060336 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.060316 5051 cpu_manager.go:414] "containerMap: removing container" podUID="82809752-7420-4fd5-b86d-74e7c1ae7432" containerName="revoke-keys" Nov 03 23:07:16.060397 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.060340 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82809752-7420-4fd5-b86d-74e7c1ae7432" containerName="revoke-keys" Nov 03 23:07:16.060451 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.060441 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="82809752-7420-4fd5-b86d-74e7c1ae7432" containerName="revoke-keys" Nov 03 23:07:16.065761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.065742 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.121732 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.121693 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt"] Nov 03 23:07:16.157498 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.157469 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4f7z\" (UniqueName: \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.157679 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.157509 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.157738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.157680 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.166506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.166478 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:16.166685 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.166626 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:07:16.258709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.258672 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4f7z\" (UniqueName: \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.259259 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.258721 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.259259 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.258829 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.259813 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.259782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.262101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.262070 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.320094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.320022 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4f7z\" (UniqueName: \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.380164 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.380138 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:16.772141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:16.772109 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt"] Nov 03 23:07:16.774267 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:07:16.774242 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8053f91_4697_4ef4_99e8_07d031b07dcf.slice/crio-4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a WatchSource:0}: Error finding container 4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a: Status 404 returned error can't find the container with id 4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a Nov 03 23:07:17.438358 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:17.438318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" event={"ID":"d8053f91-4697-4ef4-99e8-07d031b07dcf","Type":"ContainerStarted","Data":"594eb0a45a9236a53e810cf1d80d721cd6b71af63a18649cc41530a6c2fa4bf9"} Nov 03 23:07:17.438789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:17.438373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" event={"ID":"d8053f91-4697-4ef4-99e8-07d031b07dcf","Type":"ContainerStarted","Data":"4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a"} Nov 03 23:07:17.553374 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:17.552936 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" podStartSLOduration=1.552904061 podStartE2EDuration="1.552904061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:17.552779547 +0000 UTC m=+2711.290123775" watchObservedRunningTime="2026-11-03 23:07:17.552904061 +0000 UTC m=+2711.290248291" Nov 03 23:07:18.446034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:18.445982 5051 generic.go:356] "Generic (PLEG): container finished" podID="d8053f91-4697-4ef4-99e8-07d031b07dcf" containerID="594eb0a45a9236a53e810cf1d80d721cd6b71af63a18649cc41530a6c2fa4bf9" exitCode=0 Nov 03 23:07:18.446437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:18.446090 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" event={"ID":"d8053f91-4697-4ef4-99e8-07d031b07dcf","Type":"ContainerDied","Data":"594eb0a45a9236a53e810cf1d80d721cd6b71af63a18649cc41530a6c2fa4bf9"} Nov 03 23:07:19.600350 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.600328 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:19.704742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.704663 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\" (UniqueName: \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\") pod \"d8053f91-4697-4ef4-99e8-07d031b07dcf\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " Nov 03 23:07:19.704742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.704715 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\") pod \"d8053f91-4697-4ef4-99e8-07d031b07dcf\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " Nov 03 23:07:19.704979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.704785 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\") pod \"d8053f91-4697-4ef4-99e8-07d031b07dcf\" (UID: \"d8053f91-4697-4ef4-99e8-07d031b07dcf\") " Nov 03 23:07:19.705122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.705099 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca" pod "d8053f91-4697-4ef4-99e8-07d031b07dcf" (UID: "d8053f91-4697-4ef4-99e8-07d031b07dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:19.705303 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.705284 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:19.707074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.707054 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca" pod "d8053f91-4697-4ef4-99e8-07d031b07dcf" (UID: "d8053f91-4697-4ef4-99e8-07d031b07dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:19.707187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.707169 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z" pod "d8053f91-4697-4ef4-99e8-07d031b07dcf" (UID: "d8053f91-4697-4ef4-99e8-07d031b07dcf"). InnerVolumeSpecName "kube-api-access-w4f7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:19.806685 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.806649 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8053f91-4697-4ef4-99e8-07d031b07dcf-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:19.806685 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:19.806678 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4f7z\" (UniqueName: \"kubernetes.io/projected/d8053f91-4697-4ef4-99e8-07d031b07dcf-kube-api-access-w4f7z\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.472920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.472878 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" event={"ID":"d8053f91-4697-4ef4-99e8-07d031b07dcf","Type":"ContainerDied","Data":"4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a"} Nov 03 23:07:20.473141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.472950 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afa5978e8f3a81da5bb36eae5ce033765972c7fa1e6a96e3afe7cb99ec0eb1a" Nov 03 23:07:20.473141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.473075 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt" Nov 03 23:07:20.478828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.478714 5051 generic.go:356] "Generic (PLEG): container finished" podID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerID="ffcccc4c75de48c81b7d33c5f4daedb12c89293b5b52f4247adefdf3707d6ec4" exitCode=0 Nov 03 23:07:20.479296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.479112 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerDied","Data":"ffcccc4c75de48c81b7d33c5f4daedb12c89293b5b52f4247adefdf3707d6ec4"} Nov 03 23:07:20.479296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.479262 5051 scope.go:122] "RemoveContainer" containerID="e6a1df71ba436dad6997906d4ff354cc714d77c8950f4fef071ab22f85336432" Nov 03 23:07:20.518978 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.518942 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 23:07:20.615504 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615479 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615534 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615554 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615728 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615779 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.615904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.615843 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.616332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616144 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:20.616332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616166 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.616626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616461 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.616626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616584 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.616941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616671 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\") pod \"44a6af0b-f09e-4b53-be12-24c0a664efbf\" (UID: \"44a6af0b-f09e-4b53-be12-24c0a664efbf\") " Nov 03 23:07:20.616941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.616814 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.617138 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.617113 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.617343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.617257 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6af0b-f09e-4b53-be12-24c0a664efbf-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.617469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.617356 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.617469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.617374 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.617469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.617392 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.618242 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.618216 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:20.618340 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.618257 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:20.618340 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.618298 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:20.618599 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.618578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm" pod "44a6af0b-f09e-4b53-be12-24c0a664efbf" (UID: "44a6af0b-f09e-4b53-be12-24c0a664efbf"). InnerVolumeSpecName "kube-api-access-zxcnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:20.718436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.718402 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcnm\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-kube-api-access-zxcnm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.718436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.718432 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.718436 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.718443 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.718669 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.718451 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:20.718669 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:20.718460 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6af0b-f09e-4b53-be12-24c0a664efbf-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:21.494520 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:21.494490 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" Nov 03 23:07:21.494725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:21.494491 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q" event={"ID":"44a6af0b-f09e-4b53-be12-24c0a664efbf","Type":"ContainerDied","Data":"c193bc3d4bcf41470b587d6e6860c41456618eb5ac8987e122d3c1740c5e5a02"} Nov 03 23:07:21.494725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:21.494642 5051 scope.go:122] "RemoveContainer" containerID="ffcccc4c75de48c81b7d33c5f4daedb12c89293b5b52f4247adefdf3707d6ec4" Nov 03 23:07:21.623643 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:21.623607 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:07:21.676927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:21.676892 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 23:07:21.725012 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:21.724953 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eb3c33-openshift-default-8687c6c447-x7w6q"] Nov 03 23:07:22.409574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:22.409542 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" path="/var/lib/kubelet/pods/44a6af0b-f09e-4b53-be12-24c0a664efbf/volumes" Nov 03 23:07:24.884297 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:24.884266 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk"] Nov 03 23:07:24.932804 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:24.932775 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7540da-gppqk"] Nov 03 23:07:26.408874 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:26.408839 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f603ec5-91b2-41b8-986c-228831367a8a" path="/var/lib/kubelet/pods/1f603ec5-91b2-41b8-986c-228831367a8a/volumes" Nov 03 23:07:27.573555 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:27.573520 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 23:07:31.203230 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:31.203192 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx"] Nov 03 23:07:31.252493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:31.252460 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a1480-qfrcx"] Nov 03 23:07:31.778333 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:31.778295 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 23:07:31.778597 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:31.778559 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" containerID="cri-o://10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161" gracePeriod=30 Nov 03 23:07:32.411200 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.411163 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12ed289-7bcc-4f4f-a0a8-3a36bea59852" path="/var/lib/kubelet/pods/d12ed289-7bcc-4f4f-a0a8-3a36bea59852/volumes" Nov 03 23:07:32.574603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.574542 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" podUID="19988de7-6b9c-49d8-918d-c77cdf727daa" containerName="payload-processing" containerID="cri-o://c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123" gracePeriod=25 Nov 03 23:07:32.835848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.835818 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 23:07:32.884228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.884198 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\") pod \"19988de7-6b9c-49d8-918d-c77cdf727daa\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " Nov 03 23:07:32.884420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.884404 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\" (UniqueName: \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\") pod \"19988de7-6b9c-49d8-918d-c77cdf727daa\" (UID: \"19988de7-6b9c-49d8-918d-c77cdf727daa\") " Nov 03 23:07:32.884619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.884595 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume" pod "19988de7-6b9c-49d8-918d-c77cdf727daa" (UID: "19988de7-6b9c-49d8-918d-c77cdf727daa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:32.884887 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.884870 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19988de7-6b9c-49d8-918d-c77cdf727daa-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:32.886654 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.886632 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r" pod "19988de7-6b9c-49d8-918d-c77cdf727daa" (UID: "19988de7-6b9c-49d8-918d-c77cdf727daa"). InnerVolumeSpecName "kube-api-access-j2v5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:32.985839 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:32.985762 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v5r\" (UniqueName: \"kubernetes.io/projected/19988de7-6b9c-49d8-918d-c77cdf727daa-kube-api-access-j2v5r\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:33.601421 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.601381 5051 generic.go:356] "Generic (PLEG): container finished" podID="19988de7-6b9c-49d8-918d-c77cdf727daa" containerID="c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123" exitCode=0 Nov 03 23:07:33.601421 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.601427 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" Nov 03 23:07:33.601966 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.601468 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" event={"ID":"19988de7-6b9c-49d8-918d-c77cdf727daa","Type":"ContainerDied","Data":"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123"} Nov 03 23:07:33.601966 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.601522 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9" event={"ID":"19988de7-6b9c-49d8-918d-c77cdf727daa","Type":"ContainerDied","Data":"97cfa99a3337b85a0da0fb497d9e84f061e0a7ea7064fe606fea67179e2a81c9"} Nov 03 23:07:33.601966 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.601545 5051 scope.go:122] "RemoveContainer" containerID="c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123" Nov 03 23:07:33.612774 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.612753 5051 scope.go:122] "RemoveContainer" containerID="c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123" Nov 03 23:07:33.613077 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:33.613058 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123\": container with ID starting with c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123 not found: ID does not exist" containerID="c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123" Nov 03 23:07:33.613170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.613084 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123"} err="failed to get container status \"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123\": rpc error: code = NotFound desc = could not find container \"c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123\": container with ID starting with c70984ebbba3f1f47fa8ebca22beacbf30339dc6b4ec43f02d32710176f5c123 not found: ID does not exist" Nov 03 23:07:33.766013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.765962 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 23:07:33.815386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:33.815339 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-433db8-685d97b9d8-b8cj9"] Nov 03 23:07:34.409235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:34.409208 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19988de7-6b9c-49d8-918d-c77cdf727daa" path="/var/lib/kubelet/pods/19988de7-6b9c-49d8-918d-c77cdf727daa/volumes" Nov 03 23:07:37.099634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.099608 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 23:07:37.133128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133101 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133148 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133169 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133248 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133318 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133375 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.133508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133402 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133440 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133506 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133515 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.133818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133542 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\") pod \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\" (UID: \"9d68a831-daa6-43e3-b4b8-f559ca7b4902\") " Nov 03 23:07:37.133818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133684 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.133818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133735 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.134253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.133854 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:37.134324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.134258 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.134324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.134282 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.134324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.134299 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.134324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.134313 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.134556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.134328 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.135721 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.135698 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:37.135840 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.135774 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.136225 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.136202 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:37.136353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.136331 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d" pod "9d68a831-daa6-43e3-b4b8-f559ca7b4902" (UID: "9d68a831-daa6-43e3-b4b8-f559ca7b4902"). InnerVolumeSpecName "kube-api-access-kr97d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:37.235659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.235626 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.235659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.235653 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.235659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.235663 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr97d\" (UniqueName: \"kubernetes.io/projected/9d68a831-daa6-43e3-b4b8-f559ca7b4902-kube-api-access-kr97d\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.235960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.235674 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d68a831-daa6-43e3-b4b8-f559ca7b4902-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:37.633834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.633799 5051 generic.go:356] "Generic (PLEG): container finished" podID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerID="10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161" exitCode=0 Nov 03 23:07:37.634049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.633859 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" Nov 03 23:07:37.634049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.633887 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" event={"ID":"9d68a831-daa6-43e3-b4b8-f559ca7b4902","Type":"ContainerDied","Data":"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161"} Nov 03 23:07:37.634049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.633955 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj" event={"ID":"9d68a831-daa6-43e3-b4b8-f559ca7b4902","Type":"ContainerDied","Data":"d0fe964bebf7f5d735b005c8303b5772d7b92119b78c751d7f8b351b2d38a293"} Nov 03 23:07:37.634049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.633982 5051 scope.go:122] "RemoveContainer" containerID="10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161" Nov 03 23:07:37.646303 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.646288 5051 scope.go:122] "RemoveContainer" containerID="10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161" Nov 03 23:07:37.646600 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:37.646580 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161\": container with ID starting with 10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161 not found: ID does not exist" containerID="10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161" Nov 03 23:07:37.646677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.646609 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161"} err="failed to get container status \"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161\": rpc error: code = NotFound desc = could not find container \"10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161\": container with ID starting with 10e4895ddb3fbbf7596b54fb4a262ddb018d922d0ad29d780fbb7cb8ac429161 not found: ID does not exist" Nov 03 23:07:37.801920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.801885 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 23:07:37.851100 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:37.851059 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-m7rjj"] Nov 03 23:07:38.082141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:38.081314 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 23:07:38.082141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:38.081604 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" containerID="cri-o://5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de" gracePeriod=30 Nov 03 23:07:38.407800 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:38.407769 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" path="/var/lib/kubelet/pods/9d68a831-daa6-43e3-b4b8-f559ca7b4902/volumes" Nov 03 23:07:39.113401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.113362 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt"] Nov 03 23:07:39.113939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.113919 5051 cpu_manager.go:414] "containerMap: removing container" podUID="19988de7-6b9c-49d8-918d-c77cdf727daa" containerName="payload-processing" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.113941 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19988de7-6b9c-49d8-918d-c77cdf727daa" containerName="payload-processing" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.113963 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.113971 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114022 5051 cpu_manager.go:414] "containerMap: removing container" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114032 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114046 5051 cpu_manager.go:414] "containerMap: removing container" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114052 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114065 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d8053f91-4697-4ef4-99e8-07d031b07dcf" containerName="revoke-keys" Nov 03 23:07:39.114090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114071 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8053f91-4697-4ef4-99e8-07d031b07dcf" containerName="revoke-keys" Nov 03 23:07:39.114553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114216 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="19988de7-6b9c-49d8-918d-c77cdf727daa" containerName="payload-processing" Nov 03 23:07:39.114553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114230 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d68a831-daa6-43e3-b4b8-f559ca7b4902" containerName="istio-proxy" Nov 03 23:07:39.114553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114247 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114256 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a6af0b-f09e-4b53-be12-24c0a664efbf" containerName="istio-proxy" Nov 03 23:07:39.114553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.114264 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8053f91-4697-4ef4-99e8-07d031b07dcf" containerName="revoke-keys" Nov 03 23:07:39.119261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.119239 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.157221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.157188 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.157437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.157413 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvgbj\" (UniqueName: \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.157585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.157564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.190013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.179722 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt"] Nov 03 23:07:39.221387 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.221355 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:39.221553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.221397 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:07:39.258764 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.258723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.258966 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.258817 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.258966 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.258905 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvgbj\" (UniqueName: \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.259511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.259452 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.261339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.261316 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.319558 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.319523 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvgbj\" (UniqueName: \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\") pod \"maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.432792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.432704 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:39.815461 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:39.815437 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt"] Nov 03 23:07:39.817210 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:07:39.817180 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09103b97_8968_4e36_8257_29c4e5aafc62.slice/crio-7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11 WatchSource:0}: Error finding container 7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11: Status 404 returned error can't find the container with id 7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11 Nov 03 23:07:40.662015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:40.661950 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" event={"ID":"09103b97-8968-4e36-8257-29c4e5aafc62","Type":"ContainerStarted","Data":"6a965e6e5103e960edddf40e8c32a29ebcf170b6a073d7d6b987aeeb42f94ade"} Nov 03 23:07:40.662488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:40.662058 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" event={"ID":"09103b97-8968-4e36-8257-29c4e5aafc62","Type":"ContainerStarted","Data":"7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11"} Nov 03 23:07:40.781238 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:40.781118 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" podStartSLOduration=1.781103453 podStartE2EDuration="1.781103453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:40.776389975 +0000 UTC m=+2734.513734201" watchObservedRunningTime="2026-11-03 23:07:40.781103453 +0000 UTC m=+2734.518447678" Nov 03 23:07:41.686543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:41.685948 5051 generic.go:356] "Generic (PLEG): container finished" podID="09103b97-8968-4e36-8257-29c4e5aafc62" containerID="6a965e6e5103e960edddf40e8c32a29ebcf170b6a073d7d6b987aeeb42f94ade" exitCode=0 Nov 03 23:07:41.686543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:41.686278 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" event={"ID":"09103b97-8968-4e36-8257-29c4e5aafc62","Type":"ContainerDied","Data":"6a965e6e5103e960edddf40e8c32a29ebcf170b6a073d7d6b987aeeb42f94ade"} Nov 03 23:07:42.852982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.852957 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:42.905894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.905854 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\") pod \"09103b97-8968-4e36-8257-29c4e5aafc62\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " Nov 03 23:07:42.906109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.905908 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\") pod \"09103b97-8968-4e36-8257-29c4e5aafc62\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " Nov 03 23:07:42.906109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.906035 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\" (UniqueName: \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\") pod \"09103b97-8968-4e36-8257-29c4e5aafc62\" (UID: \"09103b97-8968-4e36-8257-29c4e5aafc62\") " Nov 03 23:07:42.906279 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.906255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca" pod "09103b97-8968-4e36-8257-29c4e5aafc62" (UID: "09103b97-8968-4e36-8257-29c4e5aafc62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:42.906530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.906509 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:42.908183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.908159 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca" pod "09103b97-8968-4e36-8257-29c4e5aafc62" (UID: "09103b97-8968-4e36-8257-29c4e5aafc62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:42.908288 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.908234 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj" pod "09103b97-8968-4e36-8257-29c4e5aafc62" (UID: "09103b97-8968-4e36-8257-29c4e5aafc62"). InnerVolumeSpecName "kube-api-access-wvgbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:42.958189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.958120 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt"] Nov 03 23:07:42.960929 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:42.960906 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-433db8-csxkt"] Nov 03 23:07:43.007749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.007720 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09103b97-8968-4e36-8257-29c4e5aafc62-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.007749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.007749 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvgbj\" (UniqueName: \"kubernetes.io/projected/09103b97-8968-4e36-8257-29c4e5aafc62-kube-api-access-wvgbj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.357237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.357202 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 23:07:43.412984 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.412955 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413046 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413086 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413127 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413175 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413229 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413291 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413339 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413371 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\") pod \"c31cf560-8002-451f-bc14-e2f8792597d3\" (UID: \"c31cf560-8002-451f-bc14-e2f8792597d3\") " Nov 03 23:07:43.413680 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413469 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.413875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413852 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.413875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413850 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:43.414122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.413870 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.414315 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.414288 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.414486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.414437 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.416267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.416233 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.416380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.416361 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:43.416457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.416399 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "kube-api-access-b7ssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:43.416457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.416445 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy" pod "c31cf560-8002-451f-bc14-e2f8792597d3" (UID: "c31cf560-8002-451f-bc14-e2f8792597d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:43.516415 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516378 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31cf560-8002-451f-bc14-e2f8792597d3-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516415 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516417 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516433 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516447 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31cf560-8002-451f-bc14-e2f8792597d3-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516461 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516474 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516488 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ssg\" (UniqueName: \"kubernetes.io/projected/c31cf560-8002-451f-bc14-e2f8792597d3-kube-api-access-b7ssg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.516686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.516502 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31cf560-8002-451f-bc14-e2f8792597d3-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:43.581115 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.581022 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 23:07:43.702565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.702531 5051 generic.go:356] "Generic (PLEG): container finished" podID="c31cf560-8002-451f-bc14-e2f8792597d3" containerID="5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de" exitCode=0 Nov 03 23:07:43.702773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.702606 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" Nov 03 23:07:43.702773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.702621 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerDied","Data":"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de"} Nov 03 23:07:43.702773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.702724 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq" event={"ID":"c31cf560-8002-451f-bc14-e2f8792597d3","Type":"ContainerDied","Data":"092656163c1cdc0581d8124891c8ac1b824d7e3121901bf570f1bf87269d41f4"} Nov 03 23:07:43.702773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.702762 5051 scope.go:122] "RemoveContainer" containerID="5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de" Nov 03 23:07:43.704515 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.704441 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" event={"ID":"09103b97-8968-4e36-8257-29c4e5aafc62","Type":"ContainerDied","Data":"7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11"} Nov 03 23:07:43.704515 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.704482 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f33ab570bc4bbcf85a6a0fc628c7150ac2ebc041fe532e218f8c902673cab11" Nov 03 23:07:43.704515 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.704503 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt" Nov 03 23:07:43.716441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.716412 5051 scope.go:122] "RemoveContainer" containerID="dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b" Nov 03 23:07:43.729615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.729591 5051 scope.go:122] "RemoveContainer" containerID="5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de" Nov 03 23:07:43.729932 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:43.729912 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de\": container with ID starting with 5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de not found: ID does not exist" containerID="5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de" Nov 03 23:07:43.730032 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.729941 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de"} err="failed to get container status \"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de\": rpc error: code = NotFound desc = could not find container \"5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de\": container with ID starting with 5f278634cafb8ecd457601cf641989449c3746634dcd0e9e67620707776263de not found: ID does not exist" Nov 03 23:07:43.730032 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.729957 5051 scope.go:122] "RemoveContainer" containerID="dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b" Nov 03 23:07:43.730236 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:43.730220 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b\": container with ID starting with dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b not found: ID does not exist" containerID="dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b" Nov 03 23:07:43.730293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.730239 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b"} err="failed to get container status \"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b\": rpc error: code = NotFound desc = could not find container \"dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b\": container with ID starting with dbbbfa01e620ede44d748eaf5a5b2a108180f2c124ec0e5024f86fa307e72d5b not found: ID does not exist" Nov 03 23:07:43.922840 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.922772 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 23:07:43.926824 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:43.926796 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4a1480-openshift-default-75c6f8849c-vw8pq"] Nov 03 23:07:44.408156 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:44.408118 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" path="/var/lib/kubelet/pods/c31cf560-8002-451f-bc14-e2f8792597d3/volumes" Nov 03 23:07:44.408672 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:44.408651 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8053f91-4697-4ef4-99e8-07d031b07dcf" path="/var/lib/kubelet/pods/d8053f91-4697-4ef4-99e8-07d031b07dcf/volumes" Nov 03 23:07:49.529666 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:49.529583 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 23:07:49.637013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:49.636044 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 23:07:51.508349 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.508315 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx"] Nov 03 23:07:51.509078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509049 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:07:51.509194 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509081 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:07:51.509194 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509096 5051 cpu_manager.go:414] "containerMap: removing container" podUID="09103b97-8968-4e36-8257-29c4e5aafc62" containerName="revoke-keys" Nov 03 23:07:51.509194 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509103 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09103b97-8968-4e36-8257-29c4e5aafc62" containerName="revoke-keys" Nov 03 23:07:51.509359 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509239 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:07:51.509359 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.509260 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="09103b97-8968-4e36-8257-29c4e5aafc62" containerName="revoke-keys" Nov 03 23:07:51.514203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.514187 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.568658 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.568626 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx"] Nov 03 23:07:51.613750 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.613716 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:51.613953 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.613785 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:07:51.624522 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:51.624487 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:07:51.625027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.625006 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8bb\" (UniqueName: \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.625108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.625095 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.625153 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.625129 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.725875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.725838 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8bb\" (UniqueName: \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.726107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.725886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.726107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.726074 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.726721 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.726697 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.728468 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.728446 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.786701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.786616 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8bb\" (UniqueName: \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:51.827816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:51.827783 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:52.231606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:52.231349 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx"] Nov 03 23:07:52.235732 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:07:52.235659 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb05694_2c0a_4689_a82b_c1e07bb0f45a.slice/crio-8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f WatchSource:0}: Error finding container 8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f: Status 404 returned error can't find the container with id 8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f Nov 03 23:07:52.779230 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:52.779190 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" event={"ID":"8bb05694-2c0a-4689-a82b-c1e07bb0f45a","Type":"ContainerStarted","Data":"99cb202e5b1e6f9ca6253d9fc8b9b916b700373b39712f38425b531e52f65704"} Nov 03 23:07:52.779742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:52.779276 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" event={"ID":"8bb05694-2c0a-4689-a82b-c1e07bb0f45a","Type":"ContainerStarted","Data":"8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f"} Nov 03 23:07:52.895555 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:52.895494 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" podStartSLOduration=1.895477684 podStartE2EDuration="1.895477684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:52.893336693 +0000 UTC m=+2746.630680920" watchObservedRunningTime="2026-11-03 23:07:52.895477684 +0000 UTC m=+2746.632821909" Nov 03 23:07:53.793535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:53.793483 5051 generic.go:356] "Generic (PLEG): container finished" podID="8bb05694-2c0a-4689-a82b-c1e07bb0f45a" containerID="99cb202e5b1e6f9ca6253d9fc8b9b916b700373b39712f38425b531e52f65704" exitCode=0 Nov 03 23:07:53.794214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:53.793672 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" event={"ID":"8bb05694-2c0a-4689-a82b-c1e07bb0f45a","Type":"ContainerDied","Data":"99cb202e5b1e6f9ca6253d9fc8b9b916b700373b39712f38425b531e52f65704"} Nov 03 23:07:53.948490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:53.948457 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf"] Nov 03 23:07:53.961279 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:53.961246 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfe1bd-hmvrf"] Nov 03 23:07:54.408413 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.408378 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82809752-7420-4fd5-b86d-74e7c1ae7432" path="/var/lib/kubelet/pods/82809752-7420-4fd5-b86d-74e7c1ae7432/volumes" Nov 03 23:07:54.530701 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.530662 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" podUID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" containerName="payload-processing" containerID="cri-o://87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2" gracePeriod=25 Nov 03 23:07:54.636853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.636778 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" podUID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" containerName="payload-pre-processing" containerID="cri-o://a5ce2faef493b82c658f838798e7d97f4f92f946cf16db807107d97032c0a01e" gracePeriod=25 Nov 03 23:07:54.800737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.800709 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 23:07:54.804713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.804683 5051 generic.go:356] "Generic (PLEG): container finished" podID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" containerID="87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2" exitCode=0 Nov 03 23:07:54.804857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.804806 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" event={"ID":"909d3a0e-b2e9-4696-b6f9-1596e320f82b","Type":"ContainerDied","Data":"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2"} Nov 03 23:07:54.804915 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.804854 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" event={"ID":"909d3a0e-b2e9-4696-b6f9-1596e320f82b","Type":"ContainerDied","Data":"47392480b6eae24c561496c4590f34aa2db7aeb697653edcbb1a0b4770ae7f42"} Nov 03 23:07:54.804915 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.804886 5051 scope.go:122] "RemoveContainer" containerID="87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2" Nov 03 23:07:54.805093 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.805076 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc" Nov 03 23:07:54.807469 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.807448 5051 generic.go:356] "Generic (PLEG): container finished" podID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" containerID="a5ce2faef493b82c658f838798e7d97f4f92f946cf16db807107d97032c0a01e" exitCode=0 Nov 03 23:07:54.807578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.807521 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" event={"ID":"e19dcbce-f33e-453d-a37a-d1460bafa5aa","Type":"ContainerDied","Data":"a5ce2faef493b82c658f838798e7d97f4f92f946cf16db807107d97032c0a01e"} Nov 03 23:07:54.816636 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.816604 5051 scope.go:122] "RemoveContainer" containerID="87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2" Nov 03 23:07:54.816929 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:07:54.816897 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2\": container with ID starting with 87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2 not found: ID does not exist" containerID="87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2" Nov 03 23:07:54.817020 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.816933 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2"} err="failed to get container status \"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2\": rpc error: code = NotFound desc = could not find container \"87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2\": container with ID starting with 87357d7f13ac29421acdaaa988dd9e99cc12039bca92c4f06cb2cf88933ab8d2 not found: ID does not exist" Nov 03 23:07:54.924267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.924233 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 23:07:54.967772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.967718 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\") pod \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " Nov 03 23:07:54.967956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.967801 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\" (UniqueName: \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\") pod \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\" (UID: \"909d3a0e-b2e9-4696-b6f9-1596e320f82b\") " Nov 03 23:07:54.968182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.968157 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume" pod "909d3a0e-b2e9-4696-b6f9-1596e320f82b" (UID: "909d3a0e-b2e9-4696-b6f9-1596e320f82b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:54.970727 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.970691 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8" pod "909d3a0e-b2e9-4696-b6f9-1596e320f82b" (UID: "909d3a0e-b2e9-4696-b6f9-1596e320f82b"). InnerVolumeSpecName "kube-api-access-grtm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:54.972912 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:54.972893 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:55.069307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.069273 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\" (UniqueName: \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\") pod \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " Nov 03 23:07:55.069483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.069407 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\" (UniqueName: \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\") pod \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " Nov 03 23:07:55.069483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.069430 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\") pod \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\" (UID: \"e19dcbce-f33e-453d-a37a-d1460bafa5aa\") " Nov 03 23:07:55.069627 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.069609 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\") pod \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " Nov 03 23:07:55.069677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.069650 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\") pod \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\" (UID: \"8bb05694-2c0a-4689-a82b-c1e07bb0f45a\") " Nov 03 23:07:55.070055 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.070025 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume" pod "e19dcbce-f33e-453d-a37a-d1460bafa5aa" (UID: "e19dcbce-f33e-453d-a37a-d1460bafa5aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:55.070262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.070088 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca" pod "8bb05694-2c0a-4689-a82b-c1e07bb0f45a" (UID: "8bb05694-2c0a-4689-a82b-c1e07bb0f45a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:55.070262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.070102 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909d3a0e-b2e9-4696-b6f9-1596e320f82b-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.070262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.070119 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtm8\" (UniqueName: \"kubernetes.io/projected/909d3a0e-b2e9-4696-b6f9-1596e320f82b-kube-api-access-grtm8\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.071814 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.071793 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb" pod "8bb05694-2c0a-4689-a82b-c1e07bb0f45a" (UID: "8bb05694-2c0a-4689-a82b-c1e07bb0f45a"). InnerVolumeSpecName "kube-api-access-nz8bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:55.072373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.072343 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca" pod "8bb05694-2c0a-4689-a82b-c1e07bb0f45a" (UID: "8bb05694-2c0a-4689-a82b-c1e07bb0f45a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:55.072373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.072361 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck" pod "e19dcbce-f33e-453d-a37a-d1460bafa5aa" (UID: "e19dcbce-f33e-453d-a37a-d1460bafa5aa"). InnerVolumeSpecName "kube-api-access-4m4ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:55.171659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.171625 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m4ck\" (UniqueName: \"kubernetes.io/projected/e19dcbce-f33e-453d-a37a-d1460bafa5aa-kube-api-access-4m4ck\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.171659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.171656 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19dcbce-f33e-453d-a37a-d1460bafa5aa-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.171659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.171667 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.171955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.171677 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.171955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.171685 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz8bb\" (UniqueName: \"kubernetes.io/projected/8bb05694-2c0a-4689-a82b-c1e07bb0f45a-kube-api-access-nz8bb\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:55.309307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.309221 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 23:07:55.313294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.313266 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-164c36-74dc8ddf7b-wqgvc"] Nov 03 23:07:55.815186 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.815154 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" Nov 03 23:07:55.815635 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.815198 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct" event={"ID":"e19dcbce-f33e-453d-a37a-d1460bafa5aa","Type":"ContainerDied","Data":"0dee884413a6695ef3d4b57746a771fc98ee4a819d7498e416049830351ba9ba"} Nov 03 23:07:55.815635 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.815258 5051 scope.go:122] "RemoveContainer" containerID="a5ce2faef493b82c658f838798e7d97f4f92f946cf16db807107d97032c0a01e" Nov 03 23:07:55.816906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.816880 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" Nov 03 23:07:55.817047 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.816932 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx" event={"ID":"8bb05694-2c0a-4689-a82b-c1e07bb0f45a","Type":"ContainerDied","Data":"8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f"} Nov 03 23:07:55.817047 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.816977 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8972fa8f713389b5e603ad5ec4589d0170a0e20409abd0d77543a3ceb49b085f" Nov 03 23:07:55.984101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.984060 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 23:07:55.988663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.988633 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 23:07:55.992111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:55.992085 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-164c36-84d6767d47-wn2ct"] Nov 03 23:07:56.047125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:56.047091 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6"] Nov 03 23:07:56.053733 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:56.053708 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accaaf7e-kqfw6"] Nov 03 23:07:56.409881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:56.409848 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" path="/var/lib/kubelet/pods/909d3a0e-b2e9-4696-b6f9-1596e320f82b/volumes" Nov 03 23:07:56.410413 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:56.410395 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" path="/var/lib/kubelet/pods/e19dcbce-f33e-453d-a37a-d1460bafa5aa/volumes" Nov 03 23:07:56.410780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:56.410767 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1553a0d-1253-4761-8726-5f8e173ffb6a" path="/var/lib/kubelet/pods/f1553a0d-1253-4761-8726-5f8e173ffb6a/volumes" Nov 03 23:07:58.603179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.603143 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" containerID="cri-o://cd2d8c02f2f3e92eeb20e811cdc592e0588ef187a72026b843e12f721d0d5ab0" gracePeriod=45 Nov 03 23:07:58.840681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.840645 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1027699-4263-49e9-b2db-c009f17b57cd" containerID="cd2d8c02f2f3e92eeb20e811cdc592e0588ef187a72026b843e12f721d0d5ab0" exitCode=0 Nov 03 23:07:58.840855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.840725 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" event={"ID":"b1027699-4263-49e9-b2db-c009f17b57cd","Type":"ContainerDied","Data":"cd2d8c02f2f3e92eeb20e811cdc592e0588ef187a72026b843e12f721d0d5ab0"} Nov 03 23:07:58.840855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.840786 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" event={"ID":"b1027699-4263-49e9-b2db-c009f17b57cd","Type":"ContainerDied","Data":"0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07"} Nov 03 23:07:58.840855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.840801 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5ac3fe8ad0673d6e79a63a0dbba6a996730a262ab30d6aad945c5c3c8a3f07" Nov 03 23:07:58.856982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.856924 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 23:07:58.915342 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915311 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\" (UniqueName: \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915370 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915426 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915488 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915679 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915536 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915679 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915588 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\") pod \"b1027699-4263-49e9-b2db-c009f17b57cd\" (UID: \"b1027699-4263-49e9-b2db-c009f17b57cd\") " Nov 03 23:07:58.915771 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915679 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:58.915893 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915863 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:58.916111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.915923 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:58.916203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.916150 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:58.916203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.916192 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:58.916307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.916209 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:58.918318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.918287 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:58.918432 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.918415 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:58.918500 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:58.918481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn" pod "b1027699-4263-49e9-b2db-c009f17b57cd" (UID: "b1027699-4263-49e9-b2db-c009f17b57cd"). InnerVolumeSpecName "kube-api-access-cvrwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:59.017537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:59.017503 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvrwn\" (UniqueName: \"kubernetes.io/projected/b1027699-4263-49e9-b2db-c009f17b57cd-kube-api-access-cvrwn\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.017537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:59.017532 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1027699-4263-49e9-b2db-c009f17b57cd-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.017537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:59.017541 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1027699-4263-49e9-b2db-c009f17b57cd-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.847655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:07:59.847615 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q" Nov 03 23:08:00.011090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:00.011041 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 23:08:00.060402 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:00.060370 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/distinct-w2-164c36-kserve-74ccd44b76-vk55q"] Nov 03 23:08:00.407916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:00.407884 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" path="/var/lib/kubelet/pods/b1027699-4263-49e9-b2db-c009f17b57cd/volumes" Nov 03 23:08:00.868533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:00.868498 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:08:00.869115 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:00.868731 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" containerID="cri-o://3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769" gracePeriod=30 Nov 03 23:08:01.769669 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.769634 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq"] Nov 03 23:08:01.770136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770122 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770137 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770161 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770166 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770176 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" containerName="payload-pre-processing" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770182 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" containerName="payload-pre-processing" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770189 5051 cpu_manager.go:414] "containerMap: removing container" podUID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" containerName="payload-processing" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770194 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" containerName="payload-processing" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770205 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8bb05694-2c0a-4689-a82b-c1e07bb0f45a" containerName="revoke-keys" Nov 03 23:08:01.770229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770211 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb05694-2c0a-4689-a82b-c1e07bb0f45a" containerName="revoke-keys" Nov 03 23:08:01.770717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770287 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31cf560-8002-451f-bc14-e2f8792597d3" containerName="istio-proxy" Nov 03 23:08:01.770717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770296 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1027699-4263-49e9-b2db-c009f17b57cd" containerName="main" Nov 03 23:08:01.770717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770302 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19dcbce-f33e-453d-a37a-d1460bafa5aa" containerName="payload-pre-processing" Nov 03 23:08:01.770717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770307 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="909d3a0e-b2e9-4696-b6f9-1596e320f82b" containerName="payload-processing" Nov 03 23:08:01.770717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.770313 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb05694-2c0a-4689-a82b-c1e07bb0f45a" containerName="revoke-keys" Nov 03 23:08:01.774924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.774905 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.827414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.827379 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:01.827587 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.827529 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:08:01.847351 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.847281 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.847592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.847455 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.847592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.847554 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgp7v\" (UniqueName: \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.880203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.880171 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq"] Nov 03 23:08:01.949135 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.949103 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgp7v\" (UniqueName: \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.949360 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.949173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.949360 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.949246 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.949825 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.949804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:01.951785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:01.951761 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:02.010611 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.010579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgp7v\" (UniqueName: \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\") pod \"maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:02.092533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.092440 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:02.479296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.479263 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq"] Nov 03 23:08:02.480980 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:08:02.480950 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e92b638_1714_403d_9c40_17ab296998ae.slice/crio-1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5 WatchSource:0}: Error finding container 1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5: Status 404 returned error can't find the container with id 1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5 Nov 03 23:08:02.819846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.819817 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 23:08:02.870608 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.870573 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" event={"ID":"6e92b638-1714-403d-9c40-17ab296998ae","Type":"ContainerStarted","Data":"53f5b523345883d365d3862154c553675bb610ce2f5e69484a2cfe1fff3961f1"} Nov 03 23:08:02.870808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.870634 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" event={"ID":"6e92b638-1714-403d-9c40-17ab296998ae","Type":"ContainerStarted","Data":"1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5"} Nov 03 23:08:02.925835 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:02.925795 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 23:08:03.165734 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:03.165671 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" podStartSLOduration=2.165651627 podStartE2EDuration="2.165651627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:03.164529156 +0000 UTC m=+2756.901873382" watchObservedRunningTime="2026-11-03 23:08:03.165651627 +0000 UTC m=+2756.902995852" Nov 03 23:08:03.879713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:03.879678 5051 generic.go:356] "Generic (PLEG): container finished" podID="6e92b638-1714-403d-9c40-17ab296998ae" containerID="53f5b523345883d365d3862154c553675bb610ce2f5e69484a2cfe1fff3961f1" exitCode=0 Nov 03 23:08:03.879900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:03.879774 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" event={"ID":"6e92b638-1714-403d-9c40-17ab296998ae","Type":"ContainerDied","Data":"53f5b523345883d365d3862154c553675bb610ce2f5e69484a2cfe1fff3961f1"} Nov 03 23:08:05.040144 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.040120 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:05.089372 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.089342 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\") pod \"6e92b638-1714-403d-9c40-17ab296998ae\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " Nov 03 23:08:05.089534 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.089379 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\" (UniqueName: \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\") pod \"6e92b638-1714-403d-9c40-17ab296998ae\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " Nov 03 23:08:05.089534 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.089473 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\") pod \"6e92b638-1714-403d-9c40-17ab296998ae\" (UID: \"6e92b638-1714-403d-9c40-17ab296998ae\") " Nov 03 23:08:05.089880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.089852 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca" pod "6e92b638-1714-403d-9c40-17ab296998ae" (UID: "6e92b638-1714-403d-9c40-17ab296998ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:05.091667 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.091641 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v" pod "6e92b638-1714-403d-9c40-17ab296998ae" (UID: "6e92b638-1714-403d-9c40-17ab296998ae"). InnerVolumeSpecName "kube-api-access-wgp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:05.091780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.091673 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca" pod "6e92b638-1714-403d-9c40-17ab296998ae" (UID: "6e92b638-1714-403d-9c40-17ab296998ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:05.190649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.190553 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:05.190649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.190587 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e92b638-1714-403d-9c40-17ab296998ae-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:05.190649 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.190597 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgp7v\" (UniqueName: \"kubernetes.io/projected/6e92b638-1714-403d-9c40-17ab296998ae-kube-api-access-wgp7v\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:05.895261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.895240 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" Nov 03 23:08:05.895372 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.895272 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq" event={"ID":"6e92b638-1714-403d-9c40-17ab296998ae","Type":"ContainerDied","Data":"1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5"} Nov 03 23:08:05.895372 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:05.895320 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f1d7a6f2c112346b9035371463ec3bc1cbe7f30cf5d031c85b57e243b68f8a5" Nov 03 23:08:06.163044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.161687 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:08:06.201040 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201002 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201090 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201125 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201162 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201186 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201215 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201259 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201321 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201331 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:06.201543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201387 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\") pod \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\" (UID: \"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c\") " Nov 03 23:08:06.201543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201510 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:06.201797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201552 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:06.201797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201584 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:06.201904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201836 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:06.201904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201864 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.201904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201886 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.201904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201901 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.202174 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.201916 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.203895 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.203870 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:06.204031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.203957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:06.204191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.204164 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:08:06.204300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.204193 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6" pod "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" (UID: "c2fa2c2b-eb44-4c11-be65-0d0a751ac16c"). InnerVolumeSpecName "kube-api-access-mk7q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:06.230145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.230107 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 23:08:06.232968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.232936 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 23:08:06.268362 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.268331 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 23:08:06.303665 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.303620 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.303665 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.303655 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.303925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.303675 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7q6\" (UniqueName: \"kubernetes.io/projected/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-kube-api-access-mk7q6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.303925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.303691 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.303925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.303705 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:06.904325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.904291 5051 generic.go:356] "Generic (PLEG): container finished" podID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerID="3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769" exitCode=0 Nov 03 23:08:06.904533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.904348 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" Nov 03 23:08:06.904533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.904373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerDied","Data":"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769"} Nov 03 23:08:06.904533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.904431 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht" event={"ID":"c2fa2c2b-eb44-4c11-be65-0d0a751ac16c","Type":"ContainerDied","Data":"a3d585bd3f4888c9cfff94d1aa79812a3a434b8e575df7b6ffd10c4ca4895ddb"} Nov 03 23:08:06.904533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.904457 5051 scope.go:122] "RemoveContainer" containerID="3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769" Nov 03 23:08:06.916565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.916544 5051 scope.go:122] "RemoveContainer" containerID="4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd" Nov 03 23:08:06.929816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.929796 5051 scope.go:122] "RemoveContainer" containerID="3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769" Nov 03 23:08:06.930185 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:06.930164 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769\": container with ID starting with 3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769 not found: ID does not exist" containerID="3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769" Nov 03 23:08:06.930248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.930193 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769"} err="failed to get container status \"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769\": rpc error: code = NotFound desc = could not find container \"3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769\": container with ID starting with 3e25869787be2796c803112be12e8dce7f47043e5b674de32ff9a18fc1387769 not found: ID does not exist" Nov 03 23:08:06.930248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.930213 5051 scope.go:122] "RemoveContainer" containerID="4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd" Nov 03 23:08:06.930549 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:06.930534 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd\": container with ID starting with 4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd not found: ID does not exist" containerID="4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd" Nov 03 23:08:06.930591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:06.930551 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd"} err="failed to get container status \"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd\": rpc error: code = NotFound desc = could not find container \"4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd\": container with ID starting with 4bb0a7e47d06212b001d1b851fe8fb852b8ee374e4c76d7bb4c2404f759710bd not found: ID does not exist" Nov 03 23:08:07.070488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:07.070460 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:08:07.076592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:07.076563 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cfe1bd-openshift-default-7985f9c557-5kzht"] Nov 03 23:08:07.820626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:07.820579 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" podUID="ef12284d-6252-4e47-9823-4ed240d9c58e" containerName="payload-processing" containerID="cri-o://3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52" gracePeriod=25 Nov 03 23:08:07.926918 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:07.926850 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" podUID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" containerName="payload-pre-processing" containerID="cri-o://edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a" gracePeriod=25 Nov 03 23:08:08.090571 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.090542 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 23:08:08.123066 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.123041 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\" (UniqueName: \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\") pod \"ef12284d-6252-4e47-9823-4ed240d9c58e\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " Nov 03 23:08:08.123204 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.123181 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\") pod \"ef12284d-6252-4e47-9823-4ed240d9c58e\" (UID: \"ef12284d-6252-4e47-9823-4ed240d9c58e\") " Nov 03 23:08:08.123549 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.123527 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume" pod "ef12284d-6252-4e47-9823-4ed240d9c58e" (UID: "ef12284d-6252-4e47-9823-4ed240d9c58e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:08.125502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.125475 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq" pod "ef12284d-6252-4e47-9823-4ed240d9c58e" (UID: "ef12284d-6252-4e47-9823-4ed240d9c58e"). InnerVolumeSpecName "kube-api-access-j8kjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:08.183338 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.183315 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 23:08:08.224640 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.224606 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\") pod \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " Nov 03 23:08:08.224640 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.224644 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\" (UniqueName: \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\") pod \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\" (UID: \"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e\") " Nov 03 23:08:08.224911 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.224895 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef12284d-6252-4e47-9823-4ed240d9c58e-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:08.225015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.224913 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kjq\" (UniqueName: \"kubernetes.io/projected/ef12284d-6252-4e47-9823-4ed240d9c58e-kube-api-access-j8kjq\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:08.225098 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.225069 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume" pod "dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" (UID: "dfadf4df-4b62-46c3-a3ce-46e4785e8f1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:08.226900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.226877 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x" pod "dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" (UID: "dfadf4df-4b62-46c3-a3ce-46e4785e8f1e"). InnerVolumeSpecName "kube-api-access-bbt9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:08.326491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.326420 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:08.326491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.326447 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbt9x\" (UniqueName: \"kubernetes.io/projected/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e-kube-api-access-bbt9x\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:08.410325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.410291 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" path="/var/lib/kubelet/pods/c2fa2c2b-eb44-4c11-be65-0d0a751ac16c/volumes" Nov 03 23:08:08.411287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411266 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v"] Nov 03 23:08:08.411735 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411720 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" containerName="payload-pre-processing" Nov 03 23:08:08.411780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411737 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" containerName="payload-pre-processing" Nov 03 23:08:08.411780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411761 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6e92b638-1714-403d-9c40-17ab296998ae" containerName="revoke-keys" Nov 03 23:08:08.411780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411767 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e92b638-1714-403d-9c40-17ab296998ae" containerName="revoke-keys" Nov 03 23:08:08.411956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411800 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:08.411956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411806 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:08.411956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411817 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ef12284d-6252-4e47-9823-4ed240d9c58e" containerName="payload-processing" Nov 03 23:08:08.411956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411822 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef12284d-6252-4e47-9823-4ed240d9c58e" containerName="payload-processing" Nov 03 23:08:08.411956 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411953 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:08.412227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.411966 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:08.412227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.412007 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef12284d-6252-4e47-9823-4ed240d9c58e" containerName="payload-processing" Nov 03 23:08:08.412227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.412020 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" containerName="payload-pre-processing" Nov 03 23:08:08.412227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.412036 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e92b638-1714-403d-9c40-17ab296998ae" containerName="revoke-keys" Nov 03 23:08:08.415786 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.415750 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.471427 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.471215 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:08.481556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.479206 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v"] Nov 03 23:08:08.481556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.480620 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:08:08.528694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.528651 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.528914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.528840 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.529086 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.529047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhqf\" (UniqueName: \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.630501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.630465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.630702 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.630585 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhqf\" (UniqueName: \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.630702 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.630636 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.631188 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.631162 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.633227 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.633206 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.691963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.691931 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhqf\" (UniqueName: \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\") pod \"maas-api-revoke-sub-d8e57a1b-vj62v\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.739013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.738967 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:08.920495 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.920413 5051 generic.go:356] "Generic (PLEG): container finished" podID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" containerID="edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a" exitCode=0 Nov 03 23:08:08.920495 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.920470 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" Nov 03 23:08:08.921105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.920494 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" event={"ID":"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e","Type":"ContainerDied","Data":"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a"} Nov 03 23:08:08.921105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.920547 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6" event={"ID":"dfadf4df-4b62-46c3-a3ce-46e4785e8f1e","Type":"ContainerDied","Data":"11966452b34679fddfee4bc514bf7b7e0a7c52d06f1752db59d76950f5f40105"} Nov 03 23:08:08.921105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.920573 5051 scope.go:122] "RemoveContainer" containerID="edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a" Nov 03 23:08:08.922050 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.922033 5051 generic.go:356] "Generic (PLEG): container finished" podID="ef12284d-6252-4e47-9823-4ed240d9c58e" containerID="3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52" exitCode=0 Nov 03 23:08:08.922149 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.922103 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" Nov 03 23:08:08.922149 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.922136 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" event={"ID":"ef12284d-6252-4e47-9823-4ed240d9c58e","Type":"ContainerDied","Data":"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52"} Nov 03 23:08:08.922416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.922186 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl" event={"ID":"ef12284d-6252-4e47-9823-4ed240d9c58e","Type":"ContainerDied","Data":"b5123360311ce5f8863221f086ccde787aaea63d474e272b1142678a5be45386"} Nov 03 23:08:08.931079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.931062 5051 scope.go:122] "RemoveContainer" containerID="edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a" Nov 03 23:08:08.931570 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:08.931547 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a\": container with ID starting with edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a not found: ID does not exist" containerID="edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a" Nov 03 23:08:08.931648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.931577 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a"} err="failed to get container status \"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a\": rpc error: code = NotFound desc = could not find container \"edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a\": container with ID starting with edff2ecb355aa0e8a8686702391cf9e8ab67e4ed06617ff1a1847aacdf33f85a not found: ID does not exist" Nov 03 23:08:08.931648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.931596 5051 scope.go:122] "RemoveContainer" containerID="3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52" Nov 03 23:08:08.940574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.940554 5051 scope.go:122] "RemoveContainer" containerID="3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52" Nov 03 23:08:08.940862 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:08.940842 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52\": container with ID starting with 3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52 not found: ID does not exist" containerID="3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52" Nov 03 23:08:08.940907 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:08.940868 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52"} err="failed to get container status \"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52\": rpc error: code = NotFound desc = could not find container \"3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52\": container with ID starting with 3854201aefb59a79f788be4309b34dc031a87f3ba08e5a73670df1bd42f5ff52 not found: ID does not exist" Nov 03 23:08:09.084097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.084058 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 23:08:09.088918 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.088890 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1d31f8-64f6fcf754-48jnl"] Nov 03 23:08:09.119191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.119164 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v"] Nov 03 23:08:09.120552 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:08:09.120522 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48dc75d_6227_4dd1_9431_5395c62d9a4e.slice/crio-1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f WatchSource:0}: Error finding container 1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f: Status 404 returned error can't find the container with id 1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f Nov 03 23:08:09.196493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.196425 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 23:08:09.200931 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.200905 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d31f8-f85f6f659-t2mm6"] Nov 03 23:08:09.930982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.930946 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" event={"ID":"f48dc75d-6227-4dd1-9431-5395c62d9a4e","Type":"ContainerStarted","Data":"7317068bce2d8165928723b5bace502967bef6821de9eea7b404d1ba8dc12dd9"} Nov 03 23:08:09.930982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:09.931024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" event={"ID":"f48dc75d-6227-4dd1-9431-5395c62d9a4e","Type":"ContainerStarted","Data":"1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f"} Nov 03 23:08:10.048595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:10.048545 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" podStartSLOduration=2.048532028 podStartE2EDuration="2.048532028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:10.045542672 +0000 UTC m=+2763.782886909" watchObservedRunningTime="2026-11-03 23:08:10.048532028 +0000 UTC m=+2763.785876264" Nov 03 23:08:10.407037 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:10.406978 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfadf4df-4b62-46c3-a3ce-46e4785e8f1e" path="/var/lib/kubelet/pods/dfadf4df-4b62-46c3-a3ce-46e4785e8f1e/volumes" Nov 03 23:08:10.407406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:10.407393 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef12284d-6252-4e47-9823-4ed240d9c58e" path="/var/lib/kubelet/pods/ef12284d-6252-4e47-9823-4ed240d9c58e/volumes" Nov 03 23:08:10.942077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:10.942048 5051 generic.go:356] "Generic (PLEG): container finished" podID="f48dc75d-6227-4dd1-9431-5395c62d9a4e" containerID="7317068bce2d8165928723b5bace502967bef6821de9eea7b404d1ba8dc12dd9" exitCode=0 Nov 03 23:08:10.942524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:10.942140 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" event={"ID":"f48dc75d-6227-4dd1-9431-5395c62d9a4e","Type":"ContainerDied","Data":"7317068bce2d8165928723b5bace502967bef6821de9eea7b404d1ba8dc12dd9"} Nov 03 23:08:11.010750 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.010713 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" containerID="cri-o://76bf8e34ed0d8dd7e6b2fe1a5c939d5ad4b27bfd6e2c2d11ae58b28f01f73f55" gracePeriod=45 Nov 03 23:08:11.271785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.271756 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 23:08:11.343721 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.343691 5051 scope.go:122] "RemoveContainer" containerID="cd2d8c02f2f3e92eeb20e811cdc592e0588ef187a72026b843e12f721d0d5ab0" Nov 03 23:08:11.353921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.353895 5051 scope.go:122] "RemoveContainer" containerID="76bf8e34ed0d8dd7e6b2fe1a5c939d5ad4b27bfd6e2c2d11ae58b28f01f73f55" Nov 03 23:08:11.360255 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360233 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360391 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360391 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360382 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360445 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360465 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\" (UniqueName: \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360488 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\") pod \"756749a1-27bd-499b-8eff-ddc032b61f43\" (UID: \"756749a1-27bd-499b-8eff-ddc032b61f43\") " Nov 03 23:08:11.360689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360523 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.360753 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360693 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.360753 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360740 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.360871 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.360853 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.363607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.363584 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.363796 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.363770 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:11.363910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.363865 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt" pod "756749a1-27bd-499b-8eff-ddc032b61f43" (UID: "756749a1-27bd-499b-8eff-ddc032b61f43"). InnerVolumeSpecName "kube-api-access-n8fwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:11.365598 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.365577 5051 scope.go:122] "RemoveContainer" containerID="d03c143821746f463cccc044a95d56d7f3ac3d52ccb46d8219e7d5cd461e9d1b" Nov 03 23:08:11.462121 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.462097 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462121 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.462122 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.462130 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756749a1-27bd-499b-8eff-ddc032b61f43-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.462144 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fwt\" (UniqueName: \"kubernetes.io/projected/756749a1-27bd-499b-8eff-ddc032b61f43-kube-api-access-n8fwt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.462157 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756749a1-27bd-499b-8eff-ddc032b61f43-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.948209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.948176 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" event={"ID":"756749a1-27bd-499b-8eff-ddc032b61f43","Type":"ContainerDied","Data":"76bf8e34ed0d8dd7e6b2fe1a5c939d5ad4b27bfd6e2c2d11ae58b28f01f73f55"} Nov 03 23:08:11.948728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.948226 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" event={"ID":"756749a1-27bd-499b-8eff-ddc032b61f43","Type":"ContainerDied","Data":"40710374fab5591ff5d996665ab0ed6e2c4dd3a9b640fdcc031ca5e108094ec4"} Nov 03 23:08:11.948728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:11.948280 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" Nov 03 23:08:12.089206 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.089182 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:12.168923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.168893 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\" (UniqueName: \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\") pod \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " Nov 03 23:08:12.169178 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.168946 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\") pod \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " Nov 03 23:08:12.169178 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.168967 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\") pod \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\" (UID: \"f48dc75d-6227-4dd1-9431-5395c62d9a4e\") " Nov 03 23:08:12.169394 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.169370 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca" pod "f48dc75d-6227-4dd1-9431-5395c62d9a4e" (UID: "f48dc75d-6227-4dd1-9431-5395c62d9a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:12.171327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.171306 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf" pod "f48dc75d-6227-4dd1-9431-5395c62d9a4e" (UID: "f48dc75d-6227-4dd1-9431-5395c62d9a4e"). InnerVolumeSpecName "kube-api-access-7vhqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:12.171407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.171365 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca" pod "f48dc75d-6227-4dd1-9431-5395c62d9a4e" (UID: "f48dc75d-6227-4dd1-9431-5395c62d9a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:12.179826 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.179802 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 23:08:12.183829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.183806 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw"] Nov 03 23:08:12.249255 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.249185 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/premium-w3-1d31f8-kserve-67769f7766-5prkw" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 23:08:12.270631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.270605 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhqf\" (UniqueName: \"kubernetes.io/projected/f48dc75d-6227-4dd1-9431-5395c62d9a4e-kube-api-access-7vhqf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:12.270725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.270637 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:12.270725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.270647 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48dc75d-6227-4dd1-9431-5395c62d9a4e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:12.406881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.406842 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" path="/var/lib/kubelet/pods/756749a1-27bd-499b-8eff-ddc032b61f43/volumes" Nov 03 23:08:12.956116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.956084 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" Nov 03 23:08:12.956116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.956098 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v" event={"ID":"f48dc75d-6227-4dd1-9431-5395c62d9a4e","Type":"ContainerDied","Data":"1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f"} Nov 03 23:08:12.956600 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:12.956179 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff65d1fd573354b35f6c458d36d0ab8f3b43b47da3d1767e3b72b4ddfeb596f" Nov 03 23:08:13.816890 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.816855 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9"] Nov 03 23:08:13.817545 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817530 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f48dc75d-6227-4dd1-9431-5395c62d9a4e" containerName="revoke-keys" Nov 03 23:08:13.817614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817548 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48dc75d-6227-4dd1-9431-5395c62d9a4e" containerName="revoke-keys" Nov 03 23:08:13.817614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817565 5051 cpu_manager.go:414] "containerMap: removing container" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" Nov 03 23:08:13.817614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817573 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" Nov 03 23:08:13.817614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817595 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:13.817614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817603 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fa2c2b-eb44-4c11-be65-0d0a751ac16c" containerName="istio-proxy" Nov 03 23:08:13.817797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817721 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48dc75d-6227-4dd1-9431-5395c62d9a4e" containerName="revoke-keys" Nov 03 23:08:13.817797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.817732 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="756749a1-27bd-499b-8eff-ddc032b61f43" containerName="main" Nov 03 23:08:13.821670 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.821648 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:13.868028 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.867968 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9"] Nov 03 23:08:13.913807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.913265 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htndw\" (UniqueName: \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:13.913807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.913457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:13.913807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.913536 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:13.922106 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.922072 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:08:13.922106 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:13.922094 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:08:14.014438 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.014410 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.014954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.014464 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htndw\" (UniqueName: \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.014954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.014551 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.015263 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.015238 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.016969 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.016943 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.079652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.079579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htndw\" (UniqueName: \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\") pod \"maas-api-revoke-sub-e2f9c405-5wms9\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.107614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.107583 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh"] Nov 03 23:08:14.111572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.111555 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.136600 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.136580 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:14.193066 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.193032 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh"] Nov 03 23:08:14.216113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.216083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzm5s\" (UniqueName: \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.216292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.216142 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.216292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.216231 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.317873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.317837 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.318105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.317892 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzm5s\" (UniqueName: \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.318105 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.317936 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.318541 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.318519 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.320565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.320540 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.378800 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.378771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzm5s\" (UniqueName: \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\") pod \"maas-api-revoke-sub-30c1c037-qlxfh\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.425817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.425789 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:14.556348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.556315 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9"] Nov 03 23:08:14.558356 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:08:14.558313 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11c906b_acf9_44e0_b827_6d2d4169231c.slice/crio-d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569 WatchSource:0}: Error finding container d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569: Status 404 returned error can't find the container with id d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569 Nov 03 23:08:14.815737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.815699 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh"] Nov 03 23:08:14.829461 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:08:14.829321 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30b86c4_bd6c_4332_ad05_10fedf41f310.slice/crio-4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897 WatchSource:0}: Error finding container 4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897: Status 404 returned error can't find the container with id 4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897 Nov 03 23:08:14.970557 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.970523 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" event={"ID":"c30b86c4-bd6c-4332-ad05-10fedf41f310","Type":"ContainerStarted","Data":"4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897"} Nov 03 23:08:14.972449 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.972413 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" event={"ID":"d11c906b-acf9-44e0-b827-6d2d4169231c","Type":"ContainerStarted","Data":"c4299843a9adc2f8f392c3c4ec6174e69cec91bcf6474e8e5a2e1eea6b15c4f3"} Nov 03 23:08:14.972574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:14.972461 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" event={"ID":"d11c906b-acf9-44e0-b827-6d2d4169231c","Type":"ContainerStarted","Data":"d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569"} Nov 03 23:08:15.090650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:15.090594 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" podStartSLOduration=2.090573384 podStartE2EDuration="2.090573384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:08:15.086363982 +0000 UTC m=+2768.823708208" watchObservedRunningTime="2026-11-03 23:08:15.090573384 +0000 UTC m=+2768.827917611" Nov 03 23:08:15.981050 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:15.981012 5051 generic.go:356] "Generic (PLEG): container finished" podID="d11c906b-acf9-44e0-b827-6d2d4169231c" containerID="c4299843a9adc2f8f392c3c4ec6174e69cec91bcf6474e8e5a2e1eea6b15c4f3" exitCode=0 Nov 03 23:08:15.981269 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:15.981092 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" event={"ID":"d11c906b-acf9-44e0-b827-6d2d4169231c","Type":"ContainerDied","Data":"c4299843a9adc2f8f392c3c4ec6174e69cec91bcf6474e8e5a2e1eea6b15c4f3"} Nov 03 23:08:15.983116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:15.983096 5051 generic.go:356] "Generic (PLEG): container finished" podID="c30b86c4-bd6c-4332-ad05-10fedf41f310" containerID="1e293c2076fd8057d7b4ee6f2ec7951e594a3f97c1a2db473dd9960360833711" exitCode=0 Nov 03 23:08:15.983237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:15.983167 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" event={"ID":"c30b86c4-bd6c-4332-ad05-10fedf41f310","Type":"ContainerDied","Data":"1e293c2076fd8057d7b4ee6f2ec7951e594a3f97c1a2db473dd9960360833711"} Nov 03 23:08:17.146026 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.145933 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:17.173959 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.173930 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:17.252599 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.252519 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\" (UniqueName: \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\") pod \"d11c906b-acf9-44e0-b827-6d2d4169231c\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " Nov 03 23:08:17.252771 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.252617 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\") pod \"d11c906b-acf9-44e0-b827-6d2d4169231c\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " Nov 03 23:08:17.252771 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.252660 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\") pod \"d11c906b-acf9-44e0-b827-6d2d4169231c\" (UID: \"d11c906b-acf9-44e0-b827-6d2d4169231c\") " Nov 03 23:08:17.253023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.252966 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca" pod "d11c906b-acf9-44e0-b827-6d2d4169231c" (UID: "d11c906b-acf9-44e0-b827-6d2d4169231c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:17.253173 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.253155 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.255072 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.255047 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca" pod "d11c906b-acf9-44e0-b827-6d2d4169231c" (UID: "d11c906b-acf9-44e0-b827-6d2d4169231c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:17.255158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.255076 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw" pod "d11c906b-acf9-44e0-b827-6d2d4169231c" (UID: "d11c906b-acf9-44e0-b827-6d2d4169231c"). InnerVolumeSpecName "kube-api-access-htndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:17.354091 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354060 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\") pod \"c30b86c4-bd6c-4332-ad05-10fedf41f310\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " Nov 03 23:08:17.354250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354194 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\") pod \"c30b86c4-bd6c-4332-ad05-10fedf41f310\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " Nov 03 23:08:17.354250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354247 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\" (UniqueName: \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\") pod \"c30b86c4-bd6c-4332-ad05-10fedf41f310\" (UID: \"c30b86c4-bd6c-4332-ad05-10fedf41f310\") " Nov 03 23:08:17.354472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354460 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htndw\" (UniqueName: \"kubernetes.io/projected/d11c906b-acf9-44e0-b827-6d2d4169231c-kube-api-access-htndw\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.354520 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354474 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c906b-acf9-44e0-b827-6d2d4169231c-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.354571 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.354548 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca" pod "c30b86c4-bd6c-4332-ad05-10fedf41f310" (UID: "c30b86c4-bd6c-4332-ad05-10fedf41f310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:17.356423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.356400 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca" pod "c30b86c4-bd6c-4332-ad05-10fedf41f310" (UID: "c30b86c4-bd6c-4332-ad05-10fedf41f310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:17.356510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.356449 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s" pod "c30b86c4-bd6c-4332-ad05-10fedf41f310" (UID: "c30b86c4-bd6c-4332-ad05-10fedf41f310"). InnerVolumeSpecName "kube-api-access-xzm5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:17.455236 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.455199 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.455236 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.455230 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30b86c4-bd6c-4332-ad05-10fedf41f310-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.455236 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.455244 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzm5s\" (UniqueName: \"kubernetes.io/projected/c30b86c4-bd6c-4332-ad05-10fedf41f310-kube-api-access-xzm5s\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:17.997167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.997133 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" Nov 03 23:08:17.997370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.997172 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh" event={"ID":"c30b86c4-bd6c-4332-ad05-10fedf41f310","Type":"ContainerDied","Data":"4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897"} Nov 03 23:08:17.997370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.997224 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d97bf4c84545b8b6392ed1c18182698d6dc66bdf8b2dcea0cae6590b4cab897" Nov 03 23:08:17.998900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.998878 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" Nov 03 23:08:17.999092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.998907 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9" event={"ID":"d11c906b-acf9-44e0-b827-6d2d4169231c","Type":"ContainerDied","Data":"d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569"} Nov 03 23:08:17.999092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:17.998948 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1878f2037b6fd48d301a5e818ed103954944375f390d69248fde86c34a8e569" Nov 03 23:08:21.255022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.254957 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" containerID="cri-o://3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3" gracePeriod=45 Nov 03 23:08:21.255547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.254965 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" containerID="cri-o://9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f" gracePeriod=45 Nov 03 23:08:21.292425 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.292382 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" containerID="cri-o://e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b" gracePeriod=45 Nov 03 23:08:21.639744 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.639406 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 23:08:21.640509 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:21.640342 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:08:21.650656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.650625 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 23:08:21.657005 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.656969 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 23:08:21.809034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.808901 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.809034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.808975 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809078 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809172 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\" (UniqueName: \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809189 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809215 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\" (UniqueName: \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809234 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.809285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809267 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809300 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809326 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809382 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809415 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809441 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809490 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\") pod \"7f4719b5-0280-4bab-8781-bab28aec85f3\" (UID: \"7f4719b5-0280-4bab-8781-bab28aec85f3\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809532 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809545 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809575 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809622 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\" (UniqueName: \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.809673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809655 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.810361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809726 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.810361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809731 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\") pod \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\" (UID: \"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2\") " Nov 03 23:08:21.810361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\") pod \"199474a3-6246-434d-b6ba-52689cb281a0\" (UID: \"199474a3-6246-434d-b6ba-52689cb281a0\") " Nov 03 23:08:21.810361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.809868 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.810361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810270 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810372 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810392 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810405 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810420 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810433 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810447 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810461 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.810631 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.810592 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.812415 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.812368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.813106 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.813063 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:21.813211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.813124 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:21.813211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.813123 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "kube-api-access-9n6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:21.813211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.813133 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "kube-api-access-r67j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:21.813211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.813148 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.814257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.814240 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz" pod "7f4719b5-0280-4bab-8781-bab28aec85f3" (UID: "7f4719b5-0280-4bab-8781-bab28aec85f3"). InnerVolumeSpecName "kube-api-access-sdgpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:21.814657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.814641 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs" pod "199474a3-6246-434d-b6ba-52689cb281a0" (UID: "199474a3-6246-434d-b6ba-52689cb281a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:21.814869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.814857 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm" pod "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" (UID: "46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:21.911411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911369 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4719b5-0280-4bab-8781-bab28aec85f3-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911406 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911417 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911425 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4719b5-0280-4bab-8781-bab28aec85f3-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911433 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911444 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n6dw\" (UniqueName: \"kubernetes.io/projected/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-kube-api-access-9n6dw\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911452 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199474a3-6246-434d-b6ba-52689cb281a0-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911460 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911468 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199474a3-6246-434d-b6ba-52689cb281a0-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911481 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67j6\" (UniqueName: \"kubernetes.io/projected/199474a3-6246-434d-b6ba-52689cb281a0-kube-api-access-r67j6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:21.911794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:21.911491 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdgpz\" (UniqueName: \"kubernetes.io/projected/7f4719b5-0280-4bab-8781-bab28aec85f3-kube-api-access-sdgpz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:22.029222 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.029185 5051 generic.go:356] "Generic (PLEG): container finished" podID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerID="e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b" exitCode=0 Nov 03 23:08:22.029419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.029298 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" Nov 03 23:08:22.029419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.029314 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" event={"ID":"7f4719b5-0280-4bab-8781-bab28aec85f3","Type":"ContainerDied","Data":"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b"} Nov 03 23:08:22.029419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.029355 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" event={"ID":"7f4719b5-0280-4bab-8781-bab28aec85f3","Type":"ContainerDied","Data":"77254700da84d067952c444fdfe7beb0649416c248de6dcd95ea2697b47a651a"} Nov 03 23:08:22.029419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.029378 5051 scope.go:122] "RemoveContainer" containerID="e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b" Nov 03 23:08:22.031418 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.031389 5051 generic.go:356] "Generic (PLEG): container finished" podID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerID="3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3" exitCode=0 Nov 03 23:08:22.031570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.031466 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" Nov 03 23:08:22.031570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.031514 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" event={"ID":"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2","Type":"ContainerDied","Data":"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3"} Nov 03 23:08:22.031692 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.031580 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv" event={"ID":"46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2","Type":"ContainerDied","Data":"1ee6fe79280e1ff0387f2a23aee5ef9d5d95438cf1e0700923ed49afb3e5c8cf"} Nov 03 23:08:22.045675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.045651 5051 generic.go:356] "Generic (PLEG): container finished" podID="199474a3-6246-434d-b6ba-52689cb281a0" containerID="9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f" exitCode=0 Nov 03 23:08:22.045801 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.045712 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" Nov 03 23:08:22.045801 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.045726 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" event={"ID":"199474a3-6246-434d-b6ba-52689cb281a0","Type":"ContainerDied","Data":"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f"} Nov 03 23:08:22.045910 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.045800 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" event={"ID":"199474a3-6246-434d-b6ba-52689cb281a0","Type":"ContainerDied","Data":"204b9d3b239d6349e50583b4c5601e943f6481a3fcd6bacb90ea4a1a0b186501"} Nov 03 23:08:22.047736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.047715 5051 scope.go:122] "RemoveContainer" containerID="e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b" Nov 03 23:08:22.050249 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:22.048949 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b\": container with ID starting with e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b not found: ID does not exist" containerID="e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b" Nov 03 23:08:22.050249 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.049007 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b"} err="failed to get container status \"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b\": rpc error: code = NotFound desc = could not find container \"e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b\": container with ID starting with e74e137078af7c8506f625e3e8b079f2fa7888cbc7cd8ee534fc4d99d3fcb12b not found: ID does not exist" Nov 03 23:08:22.050249 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.049033 5051 scope.go:122] "RemoveContainer" containerID="3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3" Nov 03 23:08:22.060981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.060919 5051 scope.go:122] "RemoveContainer" containerID="3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3" Nov 03 23:08:22.062086 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:22.062021 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3\": container with ID starting with 3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3 not found: ID does not exist" containerID="3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3" Nov 03 23:08:22.062086 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.062056 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3"} err="failed to get container status \"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3\": rpc error: code = NotFound desc = could not find container \"3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3\": container with ID starting with 3a4bc58c71911a1a09d9235af7511766af648d12256206747c48e6d6c78694c3 not found: ID does not exist" Nov 03 23:08:22.062086 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.062079 5051 scope.go:122] "RemoveContainer" containerID="9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f" Nov 03 23:08:22.073438 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.073417 5051 scope.go:122] "RemoveContainer" containerID="9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f" Nov 03 23:08:22.073862 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:22.073829 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f\": container with ID starting with 9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f not found: ID does not exist" containerID="9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f" Nov 03 23:08:22.073946 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.073876 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f"} err="failed to get container status \"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f\": rpc error: code = NotFound desc = could not find container \"9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f\": container with ID starting with 9f78d47949f138f68c9688b2d7f39965c005a87b1b3b382eedec4588dcdcea3f not found: ID does not exist" Nov 03 23:08:22.194342 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.194308 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 23:08:22.198441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.198415 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/distinct-w4-32449d-kserve-6997c4579c-8wqrv"] Nov 03 23:08:22.310702 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.310666 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 23:08:22.314463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.314402 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s"] Nov 03 23:08:22.407207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.407177 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" path="/var/lib/kubelet/pods/46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2/volumes" Nov 03 23:08:22.407615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.407601 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" path="/var/lib/kubelet/pods/7f4719b5-0280-4bab-8781-bab28aec85f3/volumes" Nov 03 23:08:22.422402 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.422373 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 23:08:22.426177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.426155 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666"] Nov 03 23:08:22.517238 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.517186 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/premium-w4-32449d-kserve-5977cccc6b-vj666" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:08:22.632906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:22.632863 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-32449d/simulator-w4-32449d-kserve-574ccbf495-ftt4s" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:08:24.051342 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.051313 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7"] Nov 03 23:08:24.101089 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.101059 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6244a9-n2wn7"] Nov 03 23:08:24.103632 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.103613 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj"] Nov 03 23:08:24.105903 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.105882 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105b6995-bqfzj"] Nov 03 23:08:24.407312 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.407279 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199474a3-6246-434d-b6ba-52689cb281a0" path="/var/lib/kubelet/pods/199474a3-6246-434d-b6ba-52689cb281a0/volumes" Nov 03 23:08:24.407707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.407694 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e272156-3d03-4fe9-9035-654ec582f5a2" path="/var/lib/kubelet/pods/3e272156-3d03-4fe9-9035-654ec582f5a2/volumes" Nov 03 23:08:24.408101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:24.408086 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ec840e-cc52-45d9-ae0e-11d2ce8594f6" path="/var/lib/kubelet/pods/60ec840e-cc52-45d9-ae0e-11d2ce8594f6/volumes" Nov 03 23:08:26.176807 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:26.176762 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt"] Nov 03 23:08:26.225828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:26.225794 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-164c36-nnhlt"] Nov 03 23:08:26.407033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:26.406980 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09103b97-8968-4e36-8257-29c4e5aafc62" path="/var/lib/kubelet/pods/09103b97-8968-4e36-8257-29c4e5aafc62/volumes" Nov 03 23:08:27.446021 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:27.445546 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx"] Nov 03 23:08:27.493031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:27.492970 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d31f8-x7kxx"] Nov 03 23:08:28.407378 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:28.407345 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb05694-2c0a-4689-a82b-c1e07bb0f45a" path="/var/lib/kubelet/pods/8bb05694-2c0a-4689-a82b-c1e07bb0f45a/volumes" Nov 03 23:08:33.839300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:33.839253 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 23:08:33.839785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:33.839626 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" containerID="cri-o://57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc" gracePeriod=30 Nov 03 23:08:35.070482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:35.070445 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq"] Nov 03 23:08:35.119488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:35.119453 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-32449d-s7trq"] Nov 03 23:08:36.407857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:36.407822 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e92b638-1714-403d-9c40-17ab296998ae" path="/var/lib/kubelet/pods/6e92b638-1714-403d-9c40-17ab296998ae/volumes" Nov 03 23:08:38.044816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:38.044781 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:08:38.045265 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:38.044854 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:08:39.103896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.103869 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 23:08:39.161258 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.161167 5051 generic.go:356] "Generic (PLEG): container finished" podID="b02868e4-1e87-401e-8640-71ab33a32010" containerID="57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc" exitCode=0 Nov 03 23:08:39.161258 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.161213 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" Nov 03 23:08:39.161441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.161258 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" event={"ID":"b02868e4-1e87-401e-8640-71ab33a32010","Type":"ContainerDied","Data":"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc"} Nov 03 23:08:39.161441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.161344 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk" event={"ID":"b02868e4-1e87-401e-8640-71ab33a32010","Type":"ContainerDied","Data":"bf656dca803b3596270237c93518a24b61f44b2419628ebb09c687382715fd68"} Nov 03 23:08:39.161441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.161368 5051 scope.go:122] "RemoveContainer" containerID="57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc" Nov 03 23:08:39.171607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.171556 5051 scope.go:122] "RemoveContainer" containerID="57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc" Nov 03 23:08:39.171872 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:39.171853 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc\": container with ID starting with 57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc not found: ID does not exist" containerID="57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc" Nov 03 23:08:39.171958 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.171885 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc"} err="failed to get container status \"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc\": rpc error: code = NotFound desc = could not find container \"57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc\": container with ID starting with 57e0b62e78b088bdc9171c75424cdc8060746222b01cee7f593acc8af12008fc not found: ID does not exist" Nov 03 23:08:39.188437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188410 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188506 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188539 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188603 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188698 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188637 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188698 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188669 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188761 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188795 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.188925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188810 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:39.188925 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188839 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\") pod \"b02868e4-1e87-401e-8640-71ab33a32010\" (UID: \"b02868e4-1e87-401e-8640-71ab33a32010\") " Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189068 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189087 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.188976 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189267 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189598 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189620 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02868e4-1e87-401e-8640-71ab33a32010-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189636 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189652 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.189705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.189667 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.190851 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.190828 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:08:39.190973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.190952 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:39.191075 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.191060 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "kube-api-access-j57wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:39.191433 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.191412 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token" pod "b02868e4-1e87-401e-8640-71ab33a32010" (UID: "b02868e4-1e87-401e-8640-71ab33a32010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:39.290327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.290292 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02868e4-1e87-401e-8640-71ab33a32010-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.290327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.290329 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j57wm\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-kube-api-access-j57wm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.290570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.290347 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02868e4-1e87-401e-8640-71ab33a32010-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.290570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.290358 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02868e4-1e87-401e-8640-71ab33a32010-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:39.583704 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.583671 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 23:08:39.589059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:39.589017 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-2nqsk"] Nov 03 23:08:40.407318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:40.407287 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02868e4-1e87-401e-8640-71ab33a32010" path="/var/lib/kubelet/pods/b02868e4-1e87-401e-8640-71ab33a32010/volumes" Nov 03 23:08:51.647444 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:08:51.647394 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:08:51.747181 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:51.747141 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zkrw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:08:51.747384 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:08:51.747221 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zkrw7" podUID="f6b0c86f-7342-49af-a069-ed9ff722f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:11.265955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.265916 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb"] Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266345 5051 cpu_manager.go:414] "containerMap: removing container" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266357 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266384 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266389 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266398 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c30b86c4-bd6c-4332-ad05-10fedf41f310" containerName="revoke-keys" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266403 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30b86c4-bd6c-4332-ad05-10fedf41f310" containerName="revoke-keys" Nov 03 23:09:11.266416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266418 5051 cpu_manager.go:414] "containerMap: removing container" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266423 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266430 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d11c906b-acf9-44e0-b827-6d2d4169231c" containerName="revoke-keys" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266435 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11c906b-acf9-44e0-b827-6d2d4169231c" containerName="revoke-keys" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266445 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266450 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266524 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02868e4-1e87-401e-8640-71ab33a32010" containerName="istio-proxy" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266533 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30b86c4-bd6c-4332-ad05-10fedf41f310" containerName="revoke-keys" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266541 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="46dfef3f-5d82-4eed-b46e-4c7c0c5a40c2" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266547 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11c906b-acf9-44e0-b827-6d2d4169231c" containerName="revoke-keys" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266553 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4719b5-0280-4bab-8781-bab28aec85f3" containerName="main" Nov 03 23:09:11.266653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.266559 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="199474a3-6246-434d-b6ba-52689cb281a0" containerName="main" Nov 03 23:09:11.269776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.269760 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.323806 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.323735 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:09:11.373700 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.373655 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:09:11.376892 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.376866 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb"] Nov 03 23:09:11.447332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.447290 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.447552 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.447351 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v67\" (UniqueName: \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.447552 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.447492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.497458 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.497419 5051 scope.go:122] "RemoveContainer" containerID="4965d33c8462b840c5f55dc27d3ad92bb97f3e860dfb3f8c1ddbe6837e12b6c2" Nov 03 23:09:11.507139 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.507120 5051 scope.go:122] "RemoveContainer" containerID="183f58119d21b8e3b6d6bf24293b238d747415acc75c91a2ec487290dc14388c" Nov 03 23:09:11.516489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.516467 5051 scope.go:122] "RemoveContainer" containerID="3839a76b6345e0432a49cf0df12eb6cab0928cee7a748dceee01a3d4b7bbf407" Nov 03 23:09:11.548335 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.548309 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.548440 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.548421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.548510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.548476 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v67\" (UniqueName: \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.548968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.548944 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.551128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.551101 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.609609 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.609574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v67\" (UniqueName: \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\") pod \"maas-api-revoke-sub-85237e45-t2cvb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:11.883549 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:11.883512 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:12.261720 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:12.261501 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb"] Nov 03 23:09:12.265657 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:09:12.265622 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6563b7dd_286f_4d37_bee2_efa0ce2f9dfb.slice/crio-5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34 WatchSource:0}: Error finding container 5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34: Status 404 returned error can't find the container with id 5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34 Nov 03 23:09:12.388189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:12.388150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" event={"ID":"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb","Type":"ContainerStarted","Data":"5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34"} Nov 03 23:09:13.397128 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:13.397036 5051 generic.go:356] "Generic (PLEG): container finished" podID="6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" containerID="587e304c9be2a3ee043a88f7d2069132348b11f51dca0efc170624becba449af" exitCode=0 Nov 03 23:09:13.397581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:13.397127 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" event={"ID":"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb","Type":"ContainerDied","Data":"587e304c9be2a3ee043a88f7d2069132348b11f51dca0efc170624becba449af"} Nov 03 23:09:14.548274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.548247 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:14.683205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.683115 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\" (UniqueName: \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\") pod \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " Nov 03 23:09:14.683205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.683183 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\") pod \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " Nov 03 23:09:14.683459 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.683322 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\") pod \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\" (UID: \"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb\") " Nov 03 23:09:14.683626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.683600 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca" pod "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" (UID: "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:14.683954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.683852 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:14.685501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.685476 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67" pod "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" (UID: "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb"). InnerVolumeSpecName "kube-api-access-m7v67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:14.685591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.685542 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca" pod "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" (UID: "6563b7dd-286f-4d37-bee2-efa0ce2f9dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:14.784985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.784954 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7v67\" (UniqueName: \"kubernetes.io/projected/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-kube-api-access-m7v67\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:14.784985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:14.784983 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:15.413165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:15.413081 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" Nov 03 23:09:15.413165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:15.413091 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb" event={"ID":"6563b7dd-286f-4d37-bee2-efa0ce2f9dfb","Type":"ContainerDied","Data":"5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34"} Nov 03 23:09:15.413165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:15.413149 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a41845b95fa051091989b572f14ce060186db8001604e27d285008259267b34" Nov 03 23:09:21.623082 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:09:21.623043 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:09:23.047725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:23.047689 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b"] Nov 03 23:09:23.098638 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:23.098587 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51105893-mlg9b"] Nov 03 23:09:23.102400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:23.102376 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr"] Nov 03 23:09:23.105784 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:23.105763 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f9ed975-btgxr"] Nov 03 23:09:24.406143 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:24.406115 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107c3d23-7080-4bb7-9c4f-728be09bd65a" path="/var/lib/kubelet/pods/107c3d23-7080-4bb7-9c4f-728be09bd65a/volumes" Nov 03 23:09:24.406547 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:24.406533 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7465433-bfd2-4647-bb07-ad10675cb13a" path="/var/lib/kubelet/pods/a7465433-bfd2-4647-bb07-ad10675cb13a/volumes" Nov 03 23:09:42.555785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.555751 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 23:09:42.556267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.555963 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" containerID="cri-o://a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50" gracePeriod=30 Nov 03 23:09:42.835932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.835907 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 23:09:42.960101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960067 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960430 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960406 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960548 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960449 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960548 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960529 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960598 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960716 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960656 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.960716 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.960696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\") pod \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\" (UID: \"823497da-9aa1-4fdd-a564-faf36bc4f1ad\") " Nov 03 23:09:42.961682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.961648 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:42.963312 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.963274 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:42.963585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.963563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:42.963917 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.963892 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "kube-api-access-p44wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:42.964323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.964286 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:42.964575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.964553 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:42.965700 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:42.965675 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs" pod "823497da-9aa1-4fdd-a564-faf36bc4f1ad" (UID: "823497da-9aa1-4fdd-a564-faf36bc4f1ad"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:09:43.062516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062474 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-ca-configmap\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062511 5051 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-kubeconfig\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062520 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-csr-dns-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062531 5051 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/823497da-9aa1-4fdd-a564-faf36bc4f1ad-cacerts\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062539 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44wt\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-kube-api-access-p44wt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062548 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823497da-9aa1-4fdd-a564-faf36bc4f1ad-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.062833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.062558 5051 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/823497da-9aa1-4fdd-a564-faf36bc4f1ad-local-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:09:43.592261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.592228 5051 generic.go:356] "Generic (PLEG): container finished" podID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerID="a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50" exitCode=0 Nov 03 23:09:43.592795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.592287 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" Nov 03 23:09:43.592795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.592318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" event={"ID":"823497da-9aa1-4fdd-a564-faf36bc4f1ad","Type":"ContainerDied","Data":"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50"} Nov 03 23:09:43.592795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.592373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh" event={"ID":"823497da-9aa1-4fdd-a564-faf36bc4f1ad","Type":"ContainerDied","Data":"9776a5b00d8d0a1e1ebfd8cf3dacf855993c36007b02e23429dd35da7c240bf4"} Nov 03 23:09:43.592795 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.592392 5051 scope.go:122] "RemoveContainer" containerID="a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50" Nov 03 23:09:43.604046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.604028 5051 scope.go:122] "RemoveContainer" containerID="a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50" Nov 03 23:09:43.604340 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:09:43.604318 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50\": container with ID starting with a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50 not found: ID does not exist" containerID="a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50" Nov 03 23:09:43.604418 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.604350 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50"} err="failed to get container status \"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50\": rpc error: code = NotFound desc = could not find container \"a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50\": container with ID starting with a0c36ad245aee3daad27052eda309708efb72e7c84dd9feabc8244437f48be50 not found: ID does not exist" Nov 03 23:09:43.758539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.758501 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 23:09:43.808302 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:43.808265 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vndrh"] Nov 03 23:09:44.407387 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:09:44.407354 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" path="/var/lib/kubelet/pods/823497da-9aa1-4fdd-a564-faf36bc4f1ad/volumes" Nov 03 23:09:51.623779 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:09:51.623739 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:10:11.574331 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:11.574301 5051 scope.go:122] "RemoveContainer" containerID="de42c73398b1bb37ecd247ab83b9084a3a3b1c89a045f7e4ab7de59b0500a4fa" Nov 03 23:10:11.584317 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:11.584297 5051 scope.go:122] "RemoveContainer" containerID="e0a8118f58ece671df218c1d3066812f929216ca59a287afe14fa0f145a222b6" Nov 03 23:10:11.595464 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:11.595437 5051 scope.go:122] "RemoveContainer" containerID="a1477b9ed9f4c2feec8801742699e154fcfd4182e15aa087fb717553971b3cc4" Nov 03 23:10:11.616876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:11.616784 5051 scope.go:122] "RemoveContainer" containerID="9982d4694910dbf53fb0d029de39fc00ed0ca6aebd94c583226ab665e215fd72" Nov 03 23:10:11.636365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:11.636340 5051 scope.go:122] "RemoveContainer" containerID="57ebf283d293da2f1307a81dff1f7d251bb159286542fd4fb92f0cbacf56873e" Nov 03 23:10:19.984718 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.984676 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r"] Nov 03 23:10:19.985187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985122 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" containerName="revoke-keys" Nov 03 23:10:19.985187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985134 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" containerName="revoke-keys" Nov 03 23:10:19.985187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985149 5051 cpu_manager.go:414] "containerMap: removing container" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" Nov 03 23:10:19.985187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985154 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" Nov 03 23:10:19.985343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985235 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" containerName="revoke-keys" Nov 03 23:10:19.985343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.985244 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="823497da-9aa1-4fdd-a564-faf36bc4f1ad" containerName="discovery" Nov 03 23:10:19.990133 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:19.990115 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.072912 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.072872 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.073150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.072939 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.073150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.073092 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmsz\" (UniqueName: \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.091709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.091685 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:20.091829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.091731 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:10:20.094245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.094226 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r"] Nov 03 23:10:20.174124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.174093 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.174325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.174133 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.174325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.174182 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmsz\" (UniqueName: \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.174820 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.174793 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.176712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.176689 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.235131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.235060 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmsz\" (UniqueName: \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\") pod \"maas-api-revoke-sub-c1e271a9-jfl2r\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.303244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.303217 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:20.677970 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.677946 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r"] Nov 03 23:10:20.680299 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:10:20.680272 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af5ede9_2f0f_44b9_86ed_4bfa46526e35.slice/crio-2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152 WatchSource:0}: Error finding container 2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152: Status 404 returned error can't find the container with id 2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152 Nov 03 23:10:20.874879 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:20.874848 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" event={"ID":"4af5ede9-2f0f-44b9-86ed-4bfa46526e35","Type":"ContainerStarted","Data":"2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152"} Nov 03 23:10:21.624105 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:10:21.624070 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:10:21.883604 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:21.883572 5051 generic.go:356] "Generic (PLEG): container finished" podID="4af5ede9-2f0f-44b9-86ed-4bfa46526e35" containerID="89e95db90fec5d3634e94d9faf4d5b14c517588a015f6d15f0eb4f95bea18010" exitCode=0 Nov 03 23:10:21.883829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:21.883665 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" event={"ID":"4af5ede9-2f0f-44b9-86ed-4bfa46526e35","Type":"ContainerDied","Data":"89e95db90fec5d3634e94d9faf4d5b14c517588a015f6d15f0eb4f95bea18010"} Nov 03 23:10:23.034384 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.034360 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:23.206282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.206198 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\") pod \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " Nov 03 23:10:23.206453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.206307 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\" (UniqueName: \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\") pod \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " Nov 03 23:10:23.206453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.206342 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\") pod \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\" (UID: \"4af5ede9-2f0f-44b9-86ed-4bfa46526e35\") " Nov 03 23:10:23.206628 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.206603 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca" pod "4af5ede9-2f0f-44b9-86ed-4bfa46526e35" (UID: "4af5ede9-2f0f-44b9-86ed-4bfa46526e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:23.206717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.206700 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:23.208575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.208558 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz" pod "4af5ede9-2f0f-44b9-86ed-4bfa46526e35" (UID: "4af5ede9-2f0f-44b9-86ed-4bfa46526e35"). InnerVolumeSpecName "kube-api-access-9kmsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:23.208682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.208647 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca" pod "4af5ede9-2f0f-44b9-86ed-4bfa46526e35" (UID: "4af5ede9-2f0f-44b9-86ed-4bfa46526e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:23.307553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.307519 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmsz\" (UniqueName: \"kubernetes.io/projected/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-kube-api-access-9kmsz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:23.307553 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.307549 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af5ede9-2f0f-44b9-86ed-4bfa46526e35-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:23.896928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.896894 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" Nov 03 23:10:23.897155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.896926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r" event={"ID":"4af5ede9-2f0f-44b9-86ed-4bfa46526e35","Type":"ContainerDied","Data":"2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152"} Nov 03 23:10:23.897155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:23.896984 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cee59e15dbb398c1e46ef3798c052916a48f8af68a6414b9588d1603cbd6152" Nov 03 23:10:51.290253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.290169 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t"] Nov 03 23:10:51.290742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.290596 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4af5ede9-2f0f-44b9-86ed-4bfa46526e35" containerName="revoke-keys" Nov 03 23:10:51.290742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.290607 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af5ede9-2f0f-44b9-86ed-4bfa46526e35" containerName="revoke-keys" Nov 03 23:10:51.290742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.290685 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af5ede9-2f0f-44b9-86ed-4bfa46526e35" containerName="revoke-keys" Nov 03 23:10:51.293842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.293824 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.386416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.386382 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6xp\" (UniqueName: \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.386416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.386429 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.386659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.386584 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.395190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.395158 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:10:51.395190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.395168 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:10:51.397970 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.397945 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t"] Nov 03 23:10:51.487344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.487312 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.487562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.487402 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6xp\" (UniqueName: \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.487562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.487444 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.488089 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.488062 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.489970 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.489947 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.548085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.548020 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6xp\" (UniqueName: \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\") pod \"maas-api-revoke-sub-2e1f5b5f-7k65t\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.608568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.608536 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:51.630122 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:10:51.630090 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:10:51.984532 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:51.984501 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t"] Nov 03 23:10:51.986872 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:10:51.986842 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a66be31_ca52_45ca_9635_bcc58934004f.slice/crio-463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f WatchSource:0}: Error finding container 463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f: Status 404 returned error can't find the container with id 463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f Nov 03 23:10:52.105851 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:52.105809 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" event={"ID":"8a66be31-ca52-45ca-9635-bcc58934004f","Type":"ContainerStarted","Data":"463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f"} Nov 03 23:10:53.113416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:53.113323 5051 generic.go:356] "Generic (PLEG): container finished" podID="8a66be31-ca52-45ca-9635-bcc58934004f" containerID="3c13634b1d2ddf859b0fc37f611b18c0cff2215cea43e673a86c91fcd0ba3576" exitCode=0 Nov 03 23:10:53.113810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:53.113413 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" event={"ID":"8a66be31-ca52-45ca-9635-bcc58934004f","Type":"ContainerDied","Data":"3c13634b1d2ddf859b0fc37f611b18c0cff2215cea43e673a86c91fcd0ba3576"} Nov 03 23:10:54.258787 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.258749 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:54.418198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.418112 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\") pod \"8a66be31-ca52-45ca-9635-bcc58934004f\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " Nov 03 23:10:54.418383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.418210 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\") pod \"8a66be31-ca52-45ca-9635-bcc58934004f\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " Nov 03 23:10:54.418383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.418281 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\" (UniqueName: \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\") pod \"8a66be31-ca52-45ca-9635-bcc58934004f\" (UID: \"8a66be31-ca52-45ca-9635-bcc58934004f\") " Nov 03 23:10:54.418565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.418538 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca" pod "8a66be31-ca52-45ca-9635-bcc58934004f" (UID: "8a66be31-ca52-45ca-9635-bcc58934004f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:54.420455 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.420432 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca" pod "8a66be31-ca52-45ca-9635-bcc58934004f" (UID: "8a66be31-ca52-45ca-9635-bcc58934004f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:54.420564 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.420544 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp" pod "8a66be31-ca52-45ca-9635-bcc58934004f" (UID: "8a66be31-ca52-45ca-9635-bcc58934004f"). InnerVolumeSpecName "kube-api-access-wl6xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:54.519948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.519908 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:54.519948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.519945 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a66be31-ca52-45ca-9635-bcc58934004f-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:54.519948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:54.519957 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6xp\" (UniqueName: \"kubernetes.io/projected/8a66be31-ca52-45ca-9635-bcc58934004f-kube-api-access-wl6xp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:10:55.127728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:55.127697 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" Nov 03 23:10:55.127942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:55.127735 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t" event={"ID":"8a66be31-ca52-45ca-9635-bcc58934004f","Type":"ContainerDied","Data":"463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f"} Nov 03 23:10:55.127942 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:10:55.127784 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463bd5eab0e4af46590d2caed766df8c191e016d034843abbcea671ebe718d5f" Nov 03 23:11:03.748507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.748470 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll"] Nov 03 23:11:03.749017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.748910 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8a66be31-ca52-45ca-9635-bcc58934004f" containerName="revoke-keys" Nov 03 23:11:03.749017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.748922 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a66be31-ca52-45ca-9635-bcc58934004f" containerName="revoke-keys" Nov 03 23:11:03.749017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.749019 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a66be31-ca52-45ca-9635-bcc58934004f" containerName="revoke-keys" Nov 03 23:11:03.752244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.752219 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:03.805973 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.805938 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:03.811196 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.811166 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll"] Nov 03 23:11:03.855448 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.855414 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:11:03.910499 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.910460 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:03.910690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.910508 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:03.910690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:03.910661 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkznk\" (UniqueName: \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.011705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.011617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.011705 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.011690 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.011919 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.011767 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkznk\" (UniqueName: \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.012312 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.012285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.014287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.014261 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.073025 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.072965 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkznk\" (UniqueName: \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\") pod \"maas-api-revoke-sub-ec843018-6wlll\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.365899 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.365858 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:04.771015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:04.770963 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll"] Nov 03 23:11:04.773118 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:11:04.773089 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241c5c20_47ad_40f4_a780_4c42854782df.slice/crio-e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3 WatchSource:0}: Error finding container e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3: Status 404 returned error can't find the container with id e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3 Nov 03 23:11:05.192498 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:05.192459 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" event={"ID":"241c5c20-47ad-40f4-a780-4c42854782df","Type":"ContainerStarted","Data":"8e5b3bef03c4cce9a571af27c3d844932ff8437789857fad78fe4ae8b2158ba9"} Nov 03 23:11:05.192708 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:05.192541 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" event={"ID":"241c5c20-47ad-40f4-a780-4c42854782df","Type":"ContainerStarted","Data":"e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3"} Nov 03 23:11:05.308746 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:05.308684 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" podStartSLOduration=2.308667835 podStartE2EDuration="2.308667835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:05.307246932 +0000 UTC m=+2939.044591161" watchObservedRunningTime="2026-11-03 23:11:05.308667835 +0000 UTC m=+2939.046012060" Nov 03 23:11:06.201015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:06.200950 5051 generic.go:356] "Generic (PLEG): container finished" podID="241c5c20-47ad-40f4-a780-4c42854782df" containerID="8e5b3bef03c4cce9a571af27c3d844932ff8437789857fad78fe4ae8b2158ba9" exitCode=0 Nov 03 23:11:06.201492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:06.201053 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" event={"ID":"241c5c20-47ad-40f4-a780-4c42854782df","Type":"ContainerDied","Data":"8e5b3bef03c4cce9a571af27c3d844932ff8437789857fad78fe4ae8b2158ba9"} Nov 03 23:11:07.353031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.353008 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:07.451529 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.451500 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\") pod \"241c5c20-47ad-40f4-a780-4c42854782df\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " Nov 03 23:11:07.451715 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.451548 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\") pod \"241c5c20-47ad-40f4-a780-4c42854782df\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " Nov 03 23:11:07.451715 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.451660 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\" (UniqueName: \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\") pod \"241c5c20-47ad-40f4-a780-4c42854782df\" (UID: \"241c5c20-47ad-40f4-a780-4c42854782df\") " Nov 03 23:11:07.451943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.451922 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca" pod "241c5c20-47ad-40f4-a780-4c42854782df" (UID: "241c5c20-47ad-40f4-a780-4c42854782df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:07.452109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.452079 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:07.453948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.453927 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca" pod "241c5c20-47ad-40f4-a780-4c42854782df" (UID: "241c5c20-47ad-40f4-a780-4c42854782df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:07.454044 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.453943 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk" pod "241c5c20-47ad-40f4-a780-4c42854782df" (UID: "241c5c20-47ad-40f4-a780-4c42854782df"). InnerVolumeSpecName "kube-api-access-gkznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:07.553330 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.553284 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241c5c20-47ad-40f4-a780-4c42854782df-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:07.553330 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:07.553323 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkznk\" (UniqueName: \"kubernetes.io/projected/241c5c20-47ad-40f4-a780-4c42854782df-kube-api-access-gkznk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:08.216677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:08.216645 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" event={"ID":"241c5c20-47ad-40f4-a780-4c42854782df","Type":"ContainerDied","Data":"e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3"} Nov 03 23:11:08.216677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:08.216673 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll" Nov 03 23:11:08.216923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:08.216694 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e914d8c1d4487b4d216e3ab351dbfd6c1f08874d14687f213af2a5dce17aabe3" Nov 03 23:11:11.687941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:11.687904 5051 scope.go:122] "RemoveContainer" containerID="38e024441a34b36eca369d4b415db9339d5d1fd06a979943797ce87b7ba81333" Nov 03 23:11:11.698555 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:11.698524 5051 scope.go:122] "RemoveContainer" containerID="56fb8a685b076828c0642a147de648f2d5311e0b371f6aa145700c60a12f220f" Nov 03 23:11:11.707726 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:11.707703 5051 scope.go:122] "RemoveContainer" containerID="4c694f12bd4e4f80fec3ca606f54ea0e8067cc7f6d4b29496f461103fc8d9f35" Nov 03 23:11:11.716557 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:11.716535 5051 scope.go:122] "RemoveContainer" containerID="470559ae8e6cfaafb858c0c0eb09128c8d473046c149d1e071b7a6206dfad37a" Nov 03 23:11:21.626652 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:11:21.626608 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:11:39.191560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.191518 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj"] Nov 03 23:11:39.192085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.192012 5051 cpu_manager.go:414] "containerMap: removing container" podUID="241c5c20-47ad-40f4-a780-4c42854782df" containerName="revoke-keys" Nov 03 23:11:39.192085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.192026 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241c5c20-47ad-40f4-a780-4c42854782df" containerName="revoke-keys" Nov 03 23:11:39.192233 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.192102 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="241c5c20-47ad-40f4-a780-4c42854782df" containerName="revoke-keys" Nov 03 23:11:39.195365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.195344 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.297887 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.297857 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:39.298071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.298043 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:11:39.300600 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.300577 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj"] Nov 03 23:11:39.300960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.300933 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.301124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.301105 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.301175 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.301160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzwg\" (UniqueName: \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.402269 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.402241 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.402443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.402294 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzwg\" (UniqueName: \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.402443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.402326 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.403163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.403116 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.405298 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.405269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.463084 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.463022 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzwg\" (UniqueName: \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\") pod \"maas-api-revoke-sub-046e1343-fkslj\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.509280 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.509259 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:39.890486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:39.890464 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj"] Nov 03 23:11:39.893642 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:11:39.893615 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57efb6_8a43_4de5_aba4_bc8fe4b19a8e.slice/crio-5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59 WatchSource:0}: Error finding container 5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59: Status 404 returned error can't find the container with id 5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59 Nov 03 23:11:40.446190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:40.446107 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" event={"ID":"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e","Type":"ContainerStarted","Data":"ac60c7343f17f671ab645daa295412e5f1b6522a62fe74deaeaff234d589a415"} Nov 03 23:11:40.446190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:40.446164 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" event={"ID":"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e","Type":"ContainerStarted","Data":"5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59"} Nov 03 23:11:40.561905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:40.561847 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" podStartSLOduration=1.5618322770000002 podStartE2EDuration="1.561832277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:40.560120716 +0000 UTC m=+2974.297464940" watchObservedRunningTime="2026-11-03 23:11:40.561832277 +0000 UTC m=+2974.299176504" Nov 03 23:11:41.454616 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:41.454564 5051 generic.go:356] "Generic (PLEG): container finished" podID="eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" containerID="ac60c7343f17f671ab645daa295412e5f1b6522a62fe74deaeaff234d589a415" exitCode=0 Nov 03 23:11:41.455109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:41.454637 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" event={"ID":"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e","Type":"ContainerDied","Data":"ac60c7343f17f671ab645daa295412e5f1b6522a62fe74deaeaff234d589a415"} Nov 03 23:11:42.604759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.604733 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:42.734711 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.734630 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\" (UniqueName: \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\") pod \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " Nov 03 23:11:42.734711 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.734708 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\") pod \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " Nov 03 23:11:42.734963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.734826 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\") pod \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\" (UID: \"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e\") " Nov 03 23:11:42.735107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.735075 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca" pod "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" (UID: "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:42.735210 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.735203 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:42.737158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.737129 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca" pod "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" (UID: "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:42.737522 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.737500 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg" pod "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" (UID: "eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e"). InnerVolumeSpecName "kube-api-access-7xzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:42.836061 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.836026 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:42.836061 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:42.836065 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzwg\" (UniqueName: \"kubernetes.io/projected/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e-kube-api-access-7xzwg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:43.466883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:43.466851 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" Nov 03 23:11:43.467080 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:43.466881 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj" event={"ID":"eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e","Type":"ContainerDied","Data":"5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59"} Nov 03 23:11:43.467080 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:43.466929 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6adafa76601a2f40328d6cd8a8c41f207c3b2a850b05f7338dda98acf27f59" Nov 03 23:11:47.047580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:47.047540 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr"] Nov 03 23:11:47.097375 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:47.097342 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3350a37-ktlxr"] Nov 03 23:11:47.099828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:47.099803 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj"] Nov 03 23:11:47.101817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:47.101798 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fee90a-s82xj"] Nov 03 23:11:48.406653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:48.406621 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39086f00-2822-4359-8c85-bb6d6b7bca3e" path="/var/lib/kubelet/pods/39086f00-2822-4359-8c85-bb6d6b7bca3e/volumes" Nov 03 23:11:48.407111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:48.407096 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc379af8-b66b-4daf-90ad-4d34fb00dca3" path="/var/lib/kubelet/pods/bc379af8-b66b-4daf-90ad-4d34fb00dca3/volumes" Nov 03 23:11:49.973295 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:49.973253 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2"] Nov 03 23:11:49.973911 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:49.973889 5051 cpu_manager.go:414] "containerMap: removing container" podUID="eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" containerName="revoke-keys" Nov 03 23:11:49.974054 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:49.973913 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" containerName="revoke-keys" Nov 03 23:11:49.974130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:49.974050 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" containerName="revoke-keys" Nov 03 23:11:49.978329 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:49.978307 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.034390 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.034356 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2"] Nov 03 23:11:50.080622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.080591 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:11:50.080781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.080597 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:11:50.107140 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.107105 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.107289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.107226 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.107289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.107253 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92b7\" (UniqueName: \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.208141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.208111 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.208311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.208193 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.208311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.208221 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92b7\" (UniqueName: \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.208914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.208894 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.210675 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.210652 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.269027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.268933 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92b7\" (UniqueName: \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\") pod \"maas-api-revoke-sub-d3fba056-c4sh2\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.291758 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.291734 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:50.668153 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:50.668125 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2"] Nov 03 23:11:50.670553 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:11:50.670525 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798494f5_d826_4885_bc82_626964187563.slice/crio-42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38 WatchSource:0}: Error finding container 42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38: Status 404 returned error can't find the container with id 42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38 Nov 03 23:11:51.518954 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:51.518918 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" event={"ID":"798494f5-d826-4885-bc82-626964187563","Type":"ContainerStarted","Data":"89cc80aeb436cd3434f67ab7f75486daa03814af115f99f37af51626f296b916"} Nov 03 23:11:51.519422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:51.518969 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" event={"ID":"798494f5-d826-4885-bc82-626964187563","Type":"ContainerStarted","Data":"42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38"} Nov 03 23:11:51.623287 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:11:51.623256 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:11:51.633941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:51.633886 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" podStartSLOduration=2.633871588 podStartE2EDuration="2.633871588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:51.633412362 +0000 UTC m=+2985.370756588" watchObservedRunningTime="2026-11-03 23:11:51.633871588 +0000 UTC m=+2985.371215814" Nov 03 23:11:52.525933 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:52.525895 5051 generic.go:356] "Generic (PLEG): container finished" podID="798494f5-d826-4885-bc82-626964187563" containerID="89cc80aeb436cd3434f67ab7f75486daa03814af115f99f37af51626f296b916" exitCode=0 Nov 03 23:11:52.526411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:52.525985 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" event={"ID":"798494f5-d826-4885-bc82-626964187563","Type":"ContainerDied","Data":"89cc80aeb436cd3434f67ab7f75486daa03814af115f99f37af51626f296b916"} Nov 03 23:11:54.037766 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.037725 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:11:54.085018 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.084306 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\") pod \"798494f5-d826-4885-bc82-626964187563\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " Nov 03 23:11:54.085018 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.084393 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\") pod \"798494f5-d826-4885-bc82-626964187563\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " Nov 03 23:11:54.085018 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.084469 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\" (UniqueName: \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\") pod \"798494f5-d826-4885-bc82-626964187563\" (UID: \"798494f5-d826-4885-bc82-626964187563\") " Nov 03 23:11:54.089192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.089162 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca" pod "798494f5-d826-4885-bc82-626964187563" (UID: "798494f5-d826-4885-bc82-626964187563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:11:54.097532 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.097507 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7" pod "798494f5-d826-4885-bc82-626964187563" (UID: "798494f5-d826-4885-bc82-626964187563"). InnerVolumeSpecName "kube-api-access-p92b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:11:54.115498 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.115469 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca" pod "798494f5-d826-4885-bc82-626964187563" (UID: "798494f5-d826-4885-bc82-626964187563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:11:54.189679 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.189638 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798494f5-d826-4885-bc82-626964187563-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:54.189946 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.189924 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798494f5-d826-4885-bc82-626964187563-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:54.201016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.190105 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p92b7\" (UniqueName: \"kubernetes.io/projected/798494f5-d826-4885-bc82-626964187563-kube-api-access-p92b7\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:11:54.548969 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.548937 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" event={"ID":"798494f5-d826-4885-bc82-626964187563","Type":"ContainerDied","Data":"42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38"} Nov 03 23:11:54.549182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.548985 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b82b526b3b45008fe69293b40c7eb615bc1a410a111a1c90cd23e58a84ca38" Nov 03 23:11:54.549182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:11:54.548954 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2" Nov 03 23:12:07.316223 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:07.316194 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:12:07.344320 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:07.344295 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:12:11.755082 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:11.755052 5051 scope.go:122] "RemoveContainer" containerID="cc1cbfa8f443bb138e888b0c428b7f617268aae2ae66ab1f7c8d1441b4b6213a" Nov 03 23:12:11.845785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:11.845379 5051 scope.go:122] "RemoveContainer" containerID="ec712dd108a5f3503a645f3a5281f63fd1209e54a2fb925342a24cd813737045" Nov 03 23:12:11.877402 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:11.877376 5051 scope.go:122] "RemoveContainer" containerID="dc9d3f47360f1345ced2c5d5c30f6152a3c37615ca553be9e47bfb73b20d42f7" Nov 03 23:12:11.922970 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:11.922548 5051 scope.go:122] "RemoveContainer" containerID="38d59c50102061364273bbe3891f710a7d156786465e6f25d46979a1cba62f22" Nov 03 23:12:21.623154 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:12:21.623072 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:12:24.780503 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.780465 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64"] Nov 03 23:12:24.780946 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.780928 5051 cpu_manager.go:414] "containerMap: removing container" podUID="798494f5-d826-4885-bc82-626964187563" containerName="revoke-keys" Nov 03 23:12:24.781012 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.780947 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798494f5-d826-4885-bc82-626964187563" containerName="revoke-keys" Nov 03 23:12:24.781052 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.781036 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="798494f5-d826-4885-bc82-626964187563" containerName="revoke-keys" Nov 03 23:12:24.786041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.786021 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.838207 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.838173 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:24.838481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.838260 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:12:24.874703 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.874663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bwt\" (UniqueName: \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.874884 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.874716 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.874884 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.874848 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.890159 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.890131 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64"] Nov 03 23:12:24.976151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.976106 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bwt\" (UniqueName: \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.976151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.976158 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.976357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.976198 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.976789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.976771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:24.978766 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:24.978744 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:25.037012 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:25.036935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bwt\" (UniqueName: \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\") pod \"maas-api-revoke-sub-ae594979-5jv64\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:25.100475 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:25.100449 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:25.474764 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:25.474740 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64"] Nov 03 23:12:25.477118 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:12:25.477091 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029e9988_0739_4d45_992e_691db1494fd4.slice/crio-318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0 WatchSource:0}: Error finding container 318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0: Status 404 returned error can't find the container with id 318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0 Nov 03 23:12:25.735974 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:25.735899 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" event={"ID":"029e9988-0739-4d45-992e-691db1494fd4","Type":"ContainerStarted","Data":"318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0"} Nov 03 23:12:26.743209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:26.743176 5051 generic.go:356] "Generic (PLEG): container finished" podID="029e9988-0739-4d45-992e-691db1494fd4" containerID="cd6469817146060d4f4a3a9368515310ecc7901c2122cba2a9481f4c53c19174" exitCode=0 Nov 03 23:12:26.743209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:26.743212 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" event={"ID":"029e9988-0739-4d45-992e-691db1494fd4","Type":"ContainerDied","Data":"cd6469817146060d4f4a3a9368515310ecc7901c2122cba2a9481f4c53c19174"} Nov 03 23:12:27.892004 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:27.891962 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:28.007494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.007417 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\" (UniqueName: \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\") pod \"029e9988-0739-4d45-992e-691db1494fd4\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " Nov 03 23:12:28.007494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.007475 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\") pod \"029e9988-0739-4d45-992e-691db1494fd4\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " Nov 03 23:12:28.007981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.007948 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\") pod \"029e9988-0739-4d45-992e-691db1494fd4\" (UID: \"029e9988-0739-4d45-992e-691db1494fd4\") " Nov 03 23:12:28.008549 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.008510 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca" pod "029e9988-0739-4d45-992e-691db1494fd4" (UID: "029e9988-0739-4d45-992e-691db1494fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:12:28.008738 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.008620 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029e9988-0739-4d45-992e-691db1494fd4-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:28.010309 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.010285 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca" pod "029e9988-0739-4d45-992e-691db1494fd4" (UID: "029e9988-0739-4d45-992e-691db1494fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:28.010423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.010313 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt" pod "029e9988-0739-4d45-992e-691db1494fd4" (UID: "029e9988-0739-4d45-992e-691db1494fd4"). InnerVolumeSpecName "kube-api-access-75bwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:28.110041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.109977 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bwt\" (UniqueName: \"kubernetes.io/projected/029e9988-0739-4d45-992e-691db1494fd4-kube-api-access-75bwt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:28.110041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.110038 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029e9988-0739-4d45-992e-691db1494fd4-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:28.755737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.755650 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" Nov 03 23:12:28.755737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.755676 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64" event={"ID":"029e9988-0739-4d45-992e-691db1494fd4","Type":"ContainerDied","Data":"318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0"} Nov 03 23:12:28.755737 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:28.755730 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="318b050f6c3932130ec16f1f2bdf43f6503579ec8a00dfb53468a4fcf60582f0" Nov 03 23:12:51.623210 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:12:51.623164 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:12:52.780243 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.780206 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl"] Nov 03 23:12:52.780740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.780635 5051 cpu_manager.go:414] "containerMap: removing container" podUID="029e9988-0739-4d45-992e-691db1494fd4" containerName="revoke-keys" Nov 03 23:12:52.780740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.780647 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029e9988-0739-4d45-992e-691db1494fd4" containerName="revoke-keys" Nov 03 23:12:52.780740 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.780741 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="029e9988-0739-4d45-992e-691db1494fd4" containerName="revoke-keys" Nov 03 23:12:52.784237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.784208 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:52.836760 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.836721 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:12:52.886194 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.886156 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:12:52.889442 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.889418 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl"] Nov 03 23:12:52.904451 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.904426 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:52.904562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.904487 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:52.904632 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:52.904612 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbdx\" (UniqueName: \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.006129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.006091 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.006307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.006154 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbdx\" (UniqueName: \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.006307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.006197 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.006756 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.006725 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.008678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.008660 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.067059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.066976 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbdx\" (UniqueName: \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\") pod \"maas-api-revoke-sub-a4aef7d7-7d4jl\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.097931 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.097898 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:53.479523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.479486 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl"] Nov 03 23:12:53.480431 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:12:53.480404 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0355cab2_bb9e_4f16_a7d6_00aa15912c93.slice/crio-956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e WatchSource:0}: Error finding container 956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e: Status 404 returned error can't find the container with id 956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e Nov 03 23:12:53.932875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.932835 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" event={"ID":"0355cab2-bb9e-4f16-a7d6-00aa15912c93","Type":"ContainerStarted","Data":"c5f005f7df55a447a8105cd834e26bae7e7de5b4e45c8985878079647a1a75c9"} Nov 03 23:12:53.933326 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:53.932895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" event={"ID":"0355cab2-bb9e-4f16-a7d6-00aa15912c93","Type":"ContainerStarted","Data":"956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e"} Nov 03 23:12:54.048281 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:54.048185 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" podStartSLOduration=2.048167052 podStartE2EDuration="2.048167052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:12:54.047317979 +0000 UTC m=+3047.784662205" watchObservedRunningTime="2026-11-03 23:12:54.048167052 +0000 UTC m=+3047.785511279" Nov 03 23:12:54.940257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:54.940216 5051 generic.go:356] "Generic (PLEG): container finished" podID="0355cab2-bb9e-4f16-a7d6-00aa15912c93" containerID="c5f005f7df55a447a8105cd834e26bae7e7de5b4e45c8985878079647a1a75c9" exitCode=0 Nov 03 23:12:54.940690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:54.940287 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" event={"ID":"0355cab2-bb9e-4f16-a7d6-00aa15912c93","Type":"ContainerDied","Data":"c5f005f7df55a447a8105cd834e26bae7e7de5b4e45c8985878079647a1a75c9"} Nov 03 23:12:56.083483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.083460 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:12:56.137870 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.137842 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\" (UniqueName: \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\") pod \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " Nov 03 23:12:56.138034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.137888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\") pod \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " Nov 03 23:12:56.138111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.138040 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\") pod \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\" (UID: \"0355cab2-bb9e-4f16-a7d6-00aa15912c93\") " Nov 03 23:12:56.138374 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.138350 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca" pod "0355cab2-bb9e-4f16-a7d6-00aa15912c93" (UID: "0355cab2-bb9e-4f16-a7d6-00aa15912c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:12:56.140078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.140057 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx" pod "0355cab2-bb9e-4f16-a7d6-00aa15912c93" (UID: "0355cab2-bb9e-4f16-a7d6-00aa15912c93"). InnerVolumeSpecName "kube-api-access-drbdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:12:56.140164 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.140140 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca" pod "0355cab2-bb9e-4f16-a7d6-00aa15912c93" (UID: "0355cab2-bb9e-4f16-a7d6-00aa15912c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:12:56.239834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.239759 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbdx\" (UniqueName: \"kubernetes.io/projected/0355cab2-bb9e-4f16-a7d6-00aa15912c93-kube-api-access-drbdx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:56.239834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.239790 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:56.239834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.239801 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0355cab2-bb9e-4f16-a7d6-00aa15912c93-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:12:56.954107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.954073 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" event={"ID":"0355cab2-bb9e-4f16-a7d6-00aa15912c93","Type":"ContainerDied","Data":"956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e"} Nov 03 23:12:56.954285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.954120 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956c81d7b1db8ed850d69d11971cf69319cf581683d1c99596d86fcb4467781e" Nov 03 23:12:56.954285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:12:56.954088 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl" Nov 03 23:13:11.999433 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:11.999397 5051 scope.go:122] "RemoveContainer" containerID="e2f9236e154999e03a4ad77409ee6c90b5b9d132ccb736ec69a9746adf4ac778" Nov 03 23:13:12.009409 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:12.009384 5051 scope.go:122] "RemoveContainer" containerID="d372d97e74a03521ae56edd18d9946f4315e77e5194c5a81eeb6a101047ff3a3" Nov 03 23:13:12.017746 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:12.017725 5051 scope.go:122] "RemoveContainer" containerID="05cc4acff94f20e8dceac49041010890ce506aad811e2b8310ca45ba7a3642a1" Nov 03 23:13:12.027268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:12.027237 5051 scope.go:122] "RemoveContainer" containerID="f7c442def84eee48fecbe68091fcb3531ee14e41004dd8f87e7f7657c20693a0" Nov 03 23:13:12.039296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:12.039270 5051 scope.go:122] "RemoveContainer" containerID="d535afce83aca3fe7dea523e617938f2e40e1916c06fe4084c526fa6c27bc86f" Nov 03 23:13:13.048849 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:13.048807 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v"] Nov 03 23:13:13.056711 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:13.056672 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8e57a1b-vj62v"] Nov 03 23:13:14.431027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:14.430602 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48dc75d-6227-4dd1-9431-5395c62d9a4e" path="/var/lib/kubelet/pods/f48dc75d-6227-4dd1-9431-5395c62d9a4e/volumes" Nov 03 23:13:18.047690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.047648 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh"] Nov 03 23:13:18.097249 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.097220 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c1c037-qlxfh"] Nov 03 23:13:18.099783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.099741 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9"] Nov 03 23:13:18.102635 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.102615 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f9c405-5wms9"] Nov 03 23:13:18.407192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.407157 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30b86c4-bd6c-4332-ad05-10fedf41f310" path="/var/lib/kubelet/pods/c30b86c4-bd6c-4332-ad05-10fedf41f310/volumes" Nov 03 23:13:18.407581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:18.407568 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11c906b-acf9-44e0-b827-6d2d4169231c" path="/var/lib/kubelet/pods/d11c906b-acf9-44e0-b827-6d2d4169231c/volumes" Nov 03 23:13:21.622590 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:13:21.622552 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:13:23.332416 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.332376 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm"] Nov 03 23:13:23.332980 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.332963 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0355cab2-bb9e-4f16-a7d6-00aa15912c93" containerName="revoke-keys" Nov 03 23:13:23.333056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.332983 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0355cab2-bb9e-4f16-a7d6-00aa15912c93" containerName="revoke-keys" Nov 03 23:13:23.333156 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.333144 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0355cab2-bb9e-4f16-a7d6-00aa15912c93" containerName="revoke-keys" Nov 03 23:13:23.338479 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.338456 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.392255 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.392224 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm"] Nov 03 23:13:23.422873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.422841 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9rg\" (UniqueName: \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.423192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.423165 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.423308 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.423235 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.437921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.437895 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:23.438087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.438008 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:13:23.523968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.523933 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9rg\" (UniqueName: \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.524189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.523979 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.524189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.524050 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.524665 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.524642 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.526628 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.526606 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.584952 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.584885 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9rg\" (UniqueName: \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\") pod \"maas-api-revoke-sub-79805ca9-6tqpm\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:23.651559 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:23.651526 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:24.026863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:24.026835 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm"] Nov 03 23:13:24.028620 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:13:24.028587 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d980a15_615c_443c_a6ac_e19233a00217.slice/crio-32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc WatchSource:0}: Error finding container 32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc: Status 404 returned error can't find the container with id 32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc Nov 03 23:13:24.141474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:24.141434 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" event={"ID":"3d980a15-615c-443c-a6ac-e19233a00217","Type":"ContainerStarted","Data":"32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc"} Nov 03 23:13:25.148230 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:25.148198 5051 generic.go:356] "Generic (PLEG): container finished" podID="3d980a15-615c-443c-a6ac-e19233a00217" containerID="09e0041d52072ce34e50e76a20bee7fd39607ba70f31c6e2674b1d2244aa126c" exitCode=0 Nov 03 23:13:25.148653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:25.148283 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" event={"ID":"3d980a15-615c-443c-a6ac-e19233a00217","Type":"ContainerDied","Data":"09e0041d52072ce34e50e76a20bee7fd39607ba70f31c6e2674b1d2244aa126c"} Nov 03 23:13:26.292046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.292014 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:26.349303 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.349272 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\") pod \"3d980a15-615c-443c-a6ac-e19233a00217\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " Nov 03 23:13:26.349442 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.349346 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\" (UniqueName: \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\") pod \"3d980a15-615c-443c-a6ac-e19233a00217\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " Nov 03 23:13:26.349442 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.349382 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\") pod \"3d980a15-615c-443c-a6ac-e19233a00217\" (UID: \"3d980a15-615c-443c-a6ac-e19233a00217\") " Nov 03 23:13:26.349755 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.349731 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca" pod "3d980a15-615c-443c-a6ac-e19233a00217" (UID: "3d980a15-615c-443c-a6ac-e19233a00217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:26.351447 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.351429 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca" pod "3d980a15-615c-443c-a6ac-e19233a00217" (UID: "3d980a15-615c-443c-a6ac-e19233a00217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:26.351646 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.351625 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg" pod "3d980a15-615c-443c-a6ac-e19233a00217" (UID: "3d980a15-615c-443c-a6ac-e19233a00217"). InnerVolumeSpecName "kube-api-access-9q9rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:26.450659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.450633 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9rg\" (UniqueName: \"kubernetes.io/projected/3d980a15-615c-443c-a6ac-e19233a00217-kube-api-access-9q9rg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:26.450659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.450660 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:26.450842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:26.450676 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980a15-615c-443c-a6ac-e19233a00217-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:27.163542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:27.163500 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" event={"ID":"3d980a15-615c-443c-a6ac-e19233a00217","Type":"ContainerDied","Data":"32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc"} Nov 03 23:13:27.163542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:27.163542 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm" Nov 03 23:13:27.163767 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:27.163567 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b8c831586e38928c5d7684cb7d14a0ac3e458e37a8bd56c910efe79f7aa1fc" Nov 03 23:13:47.094697 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.094617 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs"] Nov 03 23:13:47.095132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.095093 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3d980a15-615c-443c-a6ac-e19233a00217" containerName="revoke-keys" Nov 03 23:13:47.095132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.095104 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d980a15-615c-443c-a6ac-e19233a00217" containerName="revoke-keys" Nov 03 23:13:47.095222 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.095166 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d980a15-615c-443c-a6ac-e19233a00217" containerName="revoke-keys" Nov 03 23:13:47.098245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.098224 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.099191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.099166 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.099368 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.099341 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.099498 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.099409 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjh2\" (UniqueName: \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.178452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.178416 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:47.178650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.178498 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:13:47.181229 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.181201 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs"] Nov 03 23:13:47.200019 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.199970 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.200203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.200056 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjh2\" (UniqueName: \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.200203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.200104 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.200696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.200672 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.202661 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.202637 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.261113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.261081 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjh2\" (UniqueName: \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\") pod \"maas-api-revoke-sub-0b7e8edb-wlwhs\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.411642 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.411612 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:47.790950 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:47.790922 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs"] Nov 03 23:13:47.792387 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:13:47.792359 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73b0704_4e5b_4e4b_81e7_f28b9c21f8e3.slice/crio-ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512 WatchSource:0}: Error finding container ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512: Status 404 returned error can't find the container with id ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512 Nov 03 23:13:48.299905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:48.299868 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" event={"ID":"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3","Type":"ContainerStarted","Data":"78cd6425f35fe74bf2a8caa38c340b1ee711eff61cfc7d51e216b923ca7a15ab"} Nov 03 23:13:48.300345 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:48.299928 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" event={"ID":"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3","Type":"ContainerStarted","Data":"ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512"} Nov 03 23:13:48.416832 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:48.416780 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" podStartSLOduration=1.4167672869999999 podStartE2EDuration="1.416767287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:48.413737009 +0000 UTC m=+3102.151081236" watchObservedRunningTime="2026-11-03 23:13:48.416767287 +0000 UTC m=+3102.154111512" Nov 03 23:13:49.306902 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:49.306863 5051 generic.go:356] "Generic (PLEG): container finished" podID="a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" containerID="78cd6425f35fe74bf2a8caa38c340b1ee711eff61cfc7d51e216b923ca7a15ab" exitCode=0 Nov 03 23:13:49.307334 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:49.306951 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" event={"ID":"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3","Type":"ContainerDied","Data":"78cd6425f35fe74bf2a8caa38c340b1ee711eff61cfc7d51e216b923ca7a15ab"} Nov 03 23:13:50.455310 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.455286 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:50.526361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.526334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\" (UniqueName: \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\") pod \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " Nov 03 23:13:50.526507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.526374 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\") pod \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " Nov 03 23:13:50.526507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.526408 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\") pod \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\" (UID: \"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3\") " Nov 03 23:13:50.526834 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.526808 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca" pod "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" (UID: "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:50.528752 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.528730 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2" pod "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" (UID: "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3"). InnerVolumeSpecName "kube-api-access-9sjh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:50.528848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.528783 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca" pod "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" (UID: "a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:50.627646 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.627612 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:50.627646 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.627644 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:50.627857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:50.627654 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjh2\" (UniqueName: \"kubernetes.io/projected/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3-kube-api-access-9sjh2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:13:51.319575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:51.319537 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" event={"ID":"a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3","Type":"ContainerDied","Data":"ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512"} Nov 03 23:13:51.319575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:51.319556 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs" Nov 03 23:13:51.319575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:51.319584 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca259b1cedb8ea12f617c70a7f1d44015003bd16a475193c99457af02707f512" Nov 03 23:13:51.623155 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:13:51.623120 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:13:58.416815 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.416777 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7"] Nov 03 23:13:58.417501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.417268 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" containerName="revoke-keys" Nov 03 23:13:58.417501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.417282 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" containerName="revoke-keys" Nov 03 23:13:58.417501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.417346 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" containerName="revoke-keys" Nov 03 23:13:58.420656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.420636 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.496702 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.496668 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrt6\" (UniqueName: \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.496897 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.496743 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.496897 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.496776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.523699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.523671 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:13:58.523699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.523683 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:13:58.526483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.526462 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7"] Nov 03 23:13:58.597873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.597836 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrt6\" (UniqueName: \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.598079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.597928 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.598079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.597977 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.598693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.598661 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.600543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.600523 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.658540 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.658511 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrt6\" (UniqueName: \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\") pod \"maas-api-revoke-sub-3d4156f1-6r8v7\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:58.737804 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:58.737728 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:13:59.116628 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:59.116599 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7"] Nov 03 23:13:59.118561 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:13:59.118530 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35274531_0372_44f3_967a_b118788cc774.slice/crio-0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783 WatchSource:0}: Error finding container 0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783: Status 404 returned error can't find the container with id 0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783 Nov 03 23:13:59.366152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:13:59.366124 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" event={"ID":"35274531-0372-44f3-967a-b118788cc774","Type":"ContainerStarted","Data":"0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783"} Nov 03 23:14:00.376201 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:00.376165 5051 generic.go:356] "Generic (PLEG): container finished" podID="35274531-0372-44f3-967a-b118788cc774" containerID="28629d67da5a920dff6ccfc957b6a3e19c8acd377218c9e372597b06a1a3263f" exitCode=0 Nov 03 23:14:00.376639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:00.376252 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" event={"ID":"35274531-0372-44f3-967a-b118788cc774","Type":"ContainerDied","Data":"28629d67da5a920dff6ccfc957b6a3e19c8acd377218c9e372597b06a1a3263f"} Nov 03 23:14:01.932209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.932181 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:14:01.960355 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.960322 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\") pod \"35274531-0372-44f3-967a-b118788cc774\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " Nov 03 23:14:01.960940 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.960916 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\" (UniqueName: \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\") pod \"35274531-0372-44f3-967a-b118788cc774\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " Nov 03 23:14:01.961182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.961159 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\") pod \"35274531-0372-44f3-967a-b118788cc774\" (UID: \"35274531-0372-44f3-967a-b118788cc774\") " Nov 03 23:14:01.962339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.961426 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca" pod "35274531-0372-44f3-967a-b118788cc774" (UID: "35274531-0372-44f3-967a-b118788cc774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:01.976640 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.976587 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca" pod "35274531-0372-44f3-967a-b118788cc774" (UID: "35274531-0372-44f3-967a-b118788cc774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:01.977352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:01.977158 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6" pod "35274531-0372-44f3-967a-b118788cc774" (UID: "35274531-0372-44f3-967a-b118788cc774"). InnerVolumeSpecName "kube-api-access-csrt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:02.076670 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.076636 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35274531-0372-44f3-967a-b118788cc774-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:02.076979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.076958 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrt6\" (UniqueName: \"kubernetes.io/projected/35274531-0372-44f3-967a-b118788cc774-kube-api-access-csrt6\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:02.077163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.077147 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35274531-0372-44f3-967a-b118788cc774-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:02.404490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.404460 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" Nov 03 23:14:02.408430 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.408395 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7" event={"ID":"35274531-0372-44f3-967a-b118788cc774","Type":"ContainerDied","Data":"0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783"} Nov 03 23:14:02.408595 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:02.408451 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a67f04b16447202a54ebb75cbc1c6d43c91341358cc003bd33ab340aca88783" Nov 03 23:14:12.078914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:12.078865 5051 scope.go:122] "RemoveContainer" containerID="53f5b523345883d365d3862154c553675bb610ce2f5e69484a2cfe1fff3961f1" Nov 03 23:14:12.089375 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:12.089355 5051 scope.go:122] "RemoveContainer" containerID="594eb0a45a9236a53e810cf1d80d721cd6b71af63a18649cc41530a6c2fa4bf9" Nov 03 23:14:12.098603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:12.098578 5051 scope.go:122] "RemoveContainer" containerID="6a965e6e5103e960edddf40e8c32a29ebcf170b6a073d7d6b987aeeb42f94ade" Nov 03 23:14:12.107218 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:12.107194 5051 scope.go:122] "RemoveContainer" containerID="7317068bce2d8165928723b5bace502967bef6821de9eea7b404d1ba8dc12dd9" Nov 03 23:14:12.115643 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:12.115626 5051 scope.go:122] "RemoveContainer" containerID="99cb202e5b1e6f9ca6253d9fc8b9b916b700373b39712f38425b531e52f65704" Nov 03 23:14:15.047461 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:15.047426 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb"] Nov 03 23:14:15.120238 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:15.120205 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t2cvb"] Nov 03 23:14:16.407414 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:16.407377 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6563b7dd-286f-4d37-bee2-efa0ce2f9dfb" path="/var/lib/kubelet/pods/6563b7dd-286f-4d37-bee2-efa0ce2f9dfb/volumes" Nov 03 23:14:21.622483 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:14:21.622446 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:14:28.909726 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:28.909693 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7"] Nov 03 23:14:28.958814 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:28.958790 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r8v7"] Nov 03 23:14:29.032025 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.031980 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl"] Nov 03 23:14:29.032472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.032459 5051 cpu_manager.go:414] "containerMap: removing container" podUID="35274531-0372-44f3-967a-b118788cc774" containerName="revoke-keys" Nov 03 23:14:29.032516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.032474 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35274531-0372-44f3-967a-b118788cc774" containerName="revoke-keys" Nov 03 23:14:29.032565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.032556 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="35274531-0372-44f3-967a-b118788cc774" containerName="revoke-keys" Nov 03 23:14:29.037402 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.037386 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.072364 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.072338 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzvz\" (UniqueName: \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.072473 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.072371 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.072514 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.072473 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.092027 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.091981 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:14:29.092210 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.092077 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:29.094760 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.094738 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl"] Nov 03 23:14:29.173291 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.173228 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzvz\" (UniqueName: \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.173291 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.173259 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.173459 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.173326 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.173951 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.173934 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.175734 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.175715 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.234065 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.234039 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzvz\" (UniqueName: \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\") pod \"maas-api-revoke-sub-3d4156f1-thpkl\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.350306 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.350264 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:29.725907 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:29.725879 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl"] Nov 03 23:14:29.727845 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:14:29.727819 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c2abfa_7bef_477b_a881_f3241807a511.slice/crio-f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef WatchSource:0}: Error finding container f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef: Status 404 returned error can't find the container with id f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef Nov 03 23:14:30.406686 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:30.406654 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35274531-0372-44f3-967a-b118788cc774" path="/var/lib/kubelet/pods/35274531-0372-44f3-967a-b118788cc774/volumes" Nov 03 23:14:30.607179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:30.607146 5051 generic.go:356] "Generic (PLEG): container finished" podID="46c2abfa-7bef-477b-a881-f3241807a511" containerID="33d14634965bcc07e2f7973f8b1e10d9f81d47b90c3eaddf4d0aff0728e8a1b9" exitCode=0 Nov 03 23:14:30.607311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:30.607182 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" event={"ID":"46c2abfa-7bef-477b-a881-f3241807a511","Type":"ContainerDied","Data":"33d14634965bcc07e2f7973f8b1e10d9f81d47b90c3eaddf4d0aff0728e8a1b9"} Nov 03 23:14:30.607311 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:30.607230 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" event={"ID":"46c2abfa-7bef-477b-a881-f3241807a511","Type":"ContainerStarted","Data":"f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef"} Nov 03 23:14:31.740250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.740226 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:31.793450 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.793422 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\" (UniqueName: \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\") pod \"46c2abfa-7bef-477b-a881-f3241807a511\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " Nov 03 23:14:31.793593 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.793495 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\") pod \"46c2abfa-7bef-477b-a881-f3241807a511\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " Nov 03 23:14:31.793593 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.793524 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\") pod \"46c2abfa-7bef-477b-a881-f3241807a511\" (UID: \"46c2abfa-7bef-477b-a881-f3241807a511\") " Nov 03 23:14:31.793960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.793933 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca" pod "46c2abfa-7bef-477b-a881-f3241807a511" (UID: "46c2abfa-7bef-477b-a881-f3241807a511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:31.795780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.795754 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca" pod "46c2abfa-7bef-477b-a881-f3241807a511" (UID: "46c2abfa-7bef-477b-a881-f3241807a511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:31.795880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.795849 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz" pod "46c2abfa-7bef-477b-a881-f3241807a511" (UID: "46c2abfa-7bef-477b-a881-f3241807a511"). InnerVolumeSpecName "kube-api-access-rmzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:31.894465 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.894396 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:31.894465 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.894424 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c2abfa-7bef-477b-a881-f3241807a511-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:31.894465 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:31.894435 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzvz\" (UniqueName: \"kubernetes.io/projected/46c2abfa-7bef-477b-a881-f3241807a511-kube-api-access-rmzvz\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:14:32.620430 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:32.620402 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" Nov 03 23:14:32.620618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:32.620435 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl" event={"ID":"46c2abfa-7bef-477b-a881-f3241807a511","Type":"ContainerDied","Data":"f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef"} Nov 03 23:14:32.620618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:14:32.620487 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cf3f6999502045b6cbf7013e35b6e842d2992773210e1ebbef06040df373ef" Nov 03 23:14:51.623236 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:14:51.623194 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:15:00.159268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.159230 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:00.159808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.159691 5051 cpu_manager.go:414] "containerMap: removing container" podUID="46c2abfa-7bef-477b-a881-f3241807a511" containerName="revoke-keys" Nov 03 23:15:00.159808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.159705 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c2abfa-7bef-477b-a881-f3241807a511" containerName="revoke-keys" Nov 03 23:15:00.159808 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.159796 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c2abfa-7bef-477b-a881-f3241807a511" containerName="revoke-keys" Nov 03 23:15:00.162927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.162905 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.220769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.220711 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:00.255478 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.255445 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.255652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.255529 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r6z\" (UniqueName: \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.266558 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.266536 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:15:00.356927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.356888 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r6z\" (UniqueName: \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.357129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.357030 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.357637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.357618 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.418041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.417957 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r6z\" (UniqueName: \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\") pod \"maas-api-key-cleanup-29895795-fcqkm\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.481090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.481062 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:00.855125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:00.855095 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:00.856283 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:15:00.856257 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ed3ecd_f9d7_4846_a6fc_94a3508100d0.slice/crio-9fe3cf29eeb3146228d66990445f04f0116d13b8387a852380197232662f526c WatchSource:0}: Error finding container 9fe3cf29eeb3146228d66990445f04f0116d13b8387a852380197232662f526c: Status 404 returned error can't find the container with id 9fe3cf29eeb3146228d66990445f04f0116d13b8387a852380197232662f526c Nov 03 23:15:01.799915 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:01.799882 5051 generic.go:356] "Generic (PLEG): container finished" podID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerID="f8451a432d5af0481f2cbc15ad1a2a803212bf0de19689246633a104174f889b" exitCode=60 Nov 03 23:15:01.800352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:01.799926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerDied","Data":"f8451a432d5af0481f2cbc15ad1a2a803212bf0de19689246633a104174f889b"} Nov 03 23:15:01.800352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:01.800013 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerStarted","Data":"9fe3cf29eeb3146228d66990445f04f0116d13b8387a852380197232662f526c"} Nov 03 23:15:01.800352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:01.800215 5051 scope.go:122] "RemoveContainer" containerID="f8451a432d5af0481f2cbc15ad1a2a803212bf0de19689246633a104174f889b" Nov 03 23:15:02.807708 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:02.807674 5051 generic.go:356] "Generic (PLEG): container finished" podID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerID="8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9" exitCode=60 Nov 03 23:15:02.808182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:02.807757 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerDied","Data":"8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9"} Nov 03 23:15:02.808182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:02.807816 5051 scope.go:122] "RemoveContainer" containerID="f8451a432d5af0481f2cbc15ad1a2a803212bf0de19689246633a104174f889b" Nov 03 23:15:02.808182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:02.808090 5051 scope.go:122] "RemoveContainer" containerID="8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9" Nov 03 23:15:02.808367 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:15:02.808347 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-fcqkm_odh-ai-gateway-infra(22ed3ecd-f9d7-4846-a6fc-94a3508100d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" Nov 03 23:15:03.814898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:03.814864 5051 scope.go:122] "RemoveContainer" containerID="8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9" Nov 03 23:15:03.815398 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:15:03.815159 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-fcqkm_odh-ai-gateway-infra(22ed3ecd-f9d7-4846-a6fc-94a3508100d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" Nov 03 23:15:12.158408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:12.158372 5051 scope.go:122] "RemoveContainer" containerID="1e293c2076fd8057d7b4ee6f2ec7951e594a3f97c1a2db473dd9960360833711" Nov 03 23:15:12.172848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:12.172824 5051 scope.go:122] "RemoveContainer" containerID="c4299843a9adc2f8f392c3c4ec6174e69cec91bcf6474e8e5a2e1eea6b15c4f3" Nov 03 23:15:13.401285 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:13.401261 5051 scope.go:122] "RemoveContainer" containerID="8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9" Nov 03 23:15:13.914057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:13.914019 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerStarted","Data":"0eeb44d3a15c239c038f2a2981d9d8d6de30b9d8852bef50f123b9c2adcb975b"} Nov 03 23:15:14.031129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:14.031029 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" podStartSLOduration=14.03101645 podStartE2EDuration="14.03101645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:14.028022569 +0000 UTC m=+3187.765366795" watchObservedRunningTime="2026-11-03 23:15:14.03101645 +0000 UTC m=+3187.768360668" Nov 03 23:15:14.921431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:14.921393 5051 generic.go:356] "Generic (PLEG): container finished" podID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerID="0eeb44d3a15c239c038f2a2981d9d8d6de30b9d8852bef50f123b9c2adcb975b" exitCode=60 Nov 03 23:15:14.921883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:14.921474 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerDied","Data":"0eeb44d3a15c239c038f2a2981d9d8d6de30b9d8852bef50f123b9c2adcb975b"} Nov 03 23:15:14.921883 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:14.921536 5051 scope.go:122] "RemoveContainer" containerID="8add3a3607108a4f7942af0e69ae89a277875eb3335882ff70cbb0ef3aede9f9" Nov 03 23:15:14.922034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:14.921891 5051 scope.go:122] "RemoveContainer" containerID="0eeb44d3a15c239c038f2a2981d9d8d6de30b9d8852bef50f123b9c2adcb975b" Nov 03 23:15:14.922208 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:15:14.922185 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895795-fcqkm_odh-ai-gateway-infra(22ed3ecd-f9d7-4846-a6fc-94a3508100d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" Nov 03 23:15:15.045774 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:15.045742 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:16.058887 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.058864 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:16.157940 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.157912 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\") pod \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " Nov 03 23:15:16.158104 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.158091 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\" (UniqueName: \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\") pod \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\" (UID: \"22ed3ecd-f9d7-4846-a6fc-94a3508100d0\") " Nov 03 23:15:16.158264 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.158242 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca" pod "22ed3ecd-f9d7-4846-a6fc-94a3508100d0" (UID: "22ed3ecd-f9d7-4846-a6fc-94a3508100d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:16.160304 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.160275 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z" pod "22ed3ecd-f9d7-4846-a6fc-94a3508100d0" (UID: "22ed3ecd-f9d7-4846-a6fc-94a3508100d0"). InnerVolumeSpecName "kube-api-access-67r6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:16.259148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.259069 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r6z\" (UniqueName: \"kubernetes.io/projected/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-kube-api-access-67r6z\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:16.259148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.259098 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ed3ecd-f9d7-4846-a6fc-94a3508100d0-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:16.935477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.935452 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" Nov 03 23:15:16.935477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.935458 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm" event={"ID":"22ed3ecd-f9d7-4846-a6fc-94a3508100d0","Type":"ContainerDied","Data":"9fe3cf29eeb3146228d66990445f04f0116d13b8387a852380197232662f526c"} Nov 03 23:15:16.935728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:16.935512 5051 scope.go:122] "RemoveContainer" containerID="0eeb44d3a15c239c038f2a2981d9d8d6de30b9d8852bef50f123b9c2adcb975b" Nov 03 23:15:17.099111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:17.099080 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:17.149210 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:17.149178 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895795-fcqkm"] Nov 03 23:15:18.407107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:18.407078 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" path="/var/lib/kubelet/pods/22ed3ecd-f9d7-4846-a6fc-94a3508100d0/volumes" Nov 03 23:15:19.315113 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315073 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f"] Nov 03 23:15:19.315594 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315574 5051 cpu_manager.go:414] "containerMap: removing container" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315596 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315611 5051 cpu_manager.go:414] "containerMap: removing container" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315620 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315661 5051 cpu_manager.go:414] "containerMap: removing container" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315668 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315782 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.315972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.315796 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:19.320343 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.320320 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.376810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.376777 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f"] Nov 03 23:15:19.392605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.392579 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p449f\" (UniqueName: \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.392745 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.392623 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.392745 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.392718 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.422366 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.422340 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:19.422717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.422405 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:15:19.493682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.493641 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p449f\" (UniqueName: \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.493860 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.493832 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.494007 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.493962 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.494598 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.494572 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.496699 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.496677 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.554273 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.554247 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p449f\" (UniqueName: \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\") pod \"maas-api-revoke-sub-e2fc9c5b-d6s6f\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:19.633850 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:19.633826 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:20.009029 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:20.009003 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f"] Nov 03 23:15:20.011320 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:15:20.011287 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e4c097_9112_4771_9100_153be774d9ba.slice/crio-479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f WatchSource:0}: Error finding container 479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f: Status 404 returned error can't find the container with id 479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f Nov 03 23:15:20.960818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:20.960782 5051 generic.go:356] "Generic (PLEG): container finished" podID="88e4c097-9112-4771-9100-153be774d9ba" containerID="2bffcac1be1545a7ba9fc6cbe0973a45a62b71856df2394f9d2eab653b0ad55c" exitCode=0 Nov 03 23:15:20.961221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:20.960831 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" event={"ID":"88e4c097-9112-4771-9100-153be774d9ba","Type":"ContainerDied","Data":"2bffcac1be1545a7ba9fc6cbe0973a45a62b71856df2394f9d2eab653b0ad55c"} Nov 03 23:15:20.961221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:20.960885 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" event={"ID":"88e4c097-9112-4771-9100-153be774d9ba","Type":"ContainerStarted","Data":"479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f"} Nov 03 23:15:21.622781 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:15:21.622743 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:15:22.105629 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.105605 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:22.225158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.225090 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\") pod \"88e4c097-9112-4771-9100-153be774d9ba\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " Nov 03 23:15:22.225294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.225177 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\") pod \"88e4c097-9112-4771-9100-153be774d9ba\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " Nov 03 23:15:22.225294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.225238 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\" (UniqueName: \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\") pod \"88e4c097-9112-4771-9100-153be774d9ba\" (UID: \"88e4c097-9112-4771-9100-153be774d9ba\") " Nov 03 23:15:22.225572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.225544 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca" pod "88e4c097-9112-4771-9100-153be774d9ba" (UID: "88e4c097-9112-4771-9100-153be774d9ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:22.227462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.227439 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca" pod "88e4c097-9112-4771-9100-153be774d9ba" (UID: "88e4c097-9112-4771-9100-153be774d9ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:22.227736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.227717 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f" pod "88e4c097-9112-4771-9100-153be774d9ba" (UID: "88e4c097-9112-4771-9100-153be774d9ba"). InnerVolumeSpecName "kube-api-access-p449f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:22.327036 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.326973 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e4c097-9112-4771-9100-153be774d9ba-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:22.327036 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.327037 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e4c097-9112-4771-9100-153be774d9ba-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:22.327237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.327048 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p449f\" (UniqueName: \"kubernetes.io/projected/88e4c097-9112-4771-9100-153be774d9ba-kube-api-access-p449f\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:22.973440 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.973403 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" Nov 03 23:15:22.973714 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.973460 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f" event={"ID":"88e4c097-9112-4771-9100-153be774d9ba","Type":"ContainerDied","Data":"479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f"} Nov 03 23:15:22.973714 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:22.973512 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479ab94f56e66b9f174fffd8292218868a11c64f06f5aad55d718116c002c97f" Nov 03 23:15:24.046840 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:24.046809 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r"] Nov 03 23:15:24.096461 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:24.096427 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-jfl2r"] Nov 03 23:15:24.410771 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:24.410735 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af5ede9-2f0f-44b9-86ed-4bfa46526e35" path="/var/lib/kubelet/pods/4af5ede9-2f0f-44b9-86ed-4bfa46526e35/volumes" Nov 03 23:15:30.119116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.119079 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s"] Nov 03 23:15:30.119560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.119545 5051 cpu_manager.go:414] "containerMap: removing container" podUID="88e4c097-9112-4771-9100-153be774d9ba" containerName="revoke-keys" Nov 03 23:15:30.119610 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.119561 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e4c097-9112-4771-9100-153be774d9ba" containerName="revoke-keys" Nov 03 23:15:30.119663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.119654 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed3ecd-f9d7-4846-a6fc-94a3508100d0" containerName="cleanup" Nov 03 23:15:30.119700 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.119666 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e4c097-9112-4771-9100-153be774d9ba" containerName="revoke-keys" Nov 03 23:15:30.123155 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.123139 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.179770 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.179733 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s"] Nov 03 23:15:30.200864 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.200838 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzgl\" (UniqueName: \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.201033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.200895 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.201033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.201011 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.226124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.226100 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:30.226272 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.226235 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:15:30.302485 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.302449 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzgl\" (UniqueName: \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.302678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.302505 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.302678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.302537 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.303240 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.303214 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.305096 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.305078 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.363405 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.363371 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzgl\" (UniqueName: \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\") pod \"maas-api-revoke-sub-5a29efb3-c2b2s\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.435690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.435611 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:30.817325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:30.817299 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s"] Nov 03 23:15:30.820736 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:15:30.820704 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a99749_e059_488e_9132_ae97a9de7596.slice/crio-2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e WatchSource:0}: Error finding container 2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e: Status 404 returned error can't find the container with id 2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e Nov 03 23:15:31.017586 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:31.017547 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" event={"ID":"13a99749-e059-488e-9132-ae97a9de7596","Type":"ContainerStarted","Data":"2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e"} Nov 03 23:15:32.025014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:32.024906 5051 generic.go:356] "Generic (PLEG): container finished" podID="13a99749-e059-488e-9132-ae97a9de7596" containerID="3d3d75f8c7ecec476150016cb046c26a7b918f6d95effc3eba6cde0203eda2c7" exitCode=0 Nov 03 23:15:32.025408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:32.025007 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" event={"ID":"13a99749-e059-488e-9132-ae97a9de7596","Type":"ContainerDied","Data":"3d3d75f8c7ecec476150016cb046c26a7b918f6d95effc3eba6cde0203eda2c7"} Nov 03 23:15:33.220123 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.220083 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:33.335610 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.335506 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\") pod \"13a99749-e059-488e-9132-ae97a9de7596\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " Nov 03 23:15:33.335610 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.335547 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\" (UniqueName: \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\") pod \"13a99749-e059-488e-9132-ae97a9de7596\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " Nov 03 23:15:33.335878 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.335742 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\") pod \"13a99749-e059-488e-9132-ae97a9de7596\" (UID: \"13a99749-e059-488e-9132-ae97a9de7596\") " Nov 03 23:15:33.336935 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.336903 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca" pod "13a99749-e059-488e-9132-ae97a9de7596" (UID: "13a99749-e059-488e-9132-ae97a9de7596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:33.339275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.338686 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca" pod "13a99749-e059-488e-9132-ae97a9de7596" (UID: "13a99749-e059-488e-9132-ae97a9de7596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:33.342074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.342034 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl" pod "13a99749-e059-488e-9132-ae97a9de7596" (UID: "13a99749-e059-488e-9132-ae97a9de7596"). InnerVolumeSpecName "kube-api-access-4bzgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:33.446863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.446696 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a99749-e059-488e-9132-ae97a9de7596-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:33.446863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.446735 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzgl\" (UniqueName: \"kubernetes.io/projected/13a99749-e059-488e-9132-ae97a9de7596-kube-api-access-4bzgl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:33.446863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:33.446755 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a99749-e059-488e-9132-ae97a9de7596-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:34.043170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:34.043143 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" event={"ID":"13a99749-e059-488e-9132-ae97a9de7596","Type":"ContainerDied","Data":"2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e"} Nov 03 23:15:34.043446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:34.043411 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2431db4c764bfc12c96463ddc5a38b270ac8d218c791fc268e9d65f7f0c8fd6e" Nov 03 23:15:34.043446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:34.043151 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s" Nov 03 23:15:46.681870 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.681836 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl"] Nov 03 23:15:46.731017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.730969 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thpkl"] Nov 03 23:15:46.805778 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.805748 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8"] Nov 03 23:15:46.806288 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.806274 5051 cpu_manager.go:414] "containerMap: removing container" podUID="13a99749-e059-488e-9132-ae97a9de7596" containerName="revoke-keys" Nov 03 23:15:46.806340 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.806290 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a99749-e059-488e-9132-ae97a9de7596" containerName="revoke-keys" Nov 03 23:15:46.806386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.806377 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a99749-e059-488e-9132-ae97a9de7596" containerName="revoke-keys" Nov 03 23:15:46.809526 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.809511 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:46.864522 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.864495 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:46.864700 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.864576 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:15:46.869140 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.869114 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8"] Nov 03 23:15:46.994248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.994146 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7s9\" (UniqueName: \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:46.994248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.994202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:46.994470 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:46.994402 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.095295 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.095260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.095476 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.095361 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7s9\" (UniqueName: \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.095476 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.095386 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.096003 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.095954 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.098489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.098464 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.156423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.156392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7s9\" (UniqueName: \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\") pod \"maas-api-revoke-sub-3d4156f1-44zx8\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.422786 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.422751 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:47.801401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:47.801375 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8"] Nov 03 23:15:47.803586 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:15:47.803563 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e73d6c2_2c6c_4193_8892_a0cc5a38208d.slice/crio-503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29 WatchSource:0}: Error finding container 503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29: Status 404 returned error can't find the container with id 503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29 Nov 03 23:15:48.133690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:48.133651 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" event={"ID":"2e73d6c2-2c6c-4193-8892-a0cc5a38208d","Type":"ContainerStarted","Data":"0e89ad10a630e816a5baa1918eb411366814f889da3cbad162ce628c6920f51d"} Nov 03 23:15:48.133849 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:48.133721 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" event={"ID":"2e73d6c2-2c6c-4193-8892-a0cc5a38208d","Type":"ContainerStarted","Data":"503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29"} Nov 03 23:15:48.249510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:48.249459 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" podStartSLOduration=2.24943834 podStartE2EDuration="2.24943834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:48.247954442 +0000 UTC m=+3221.985298667" watchObservedRunningTime="2026-11-03 23:15:48.24943834 +0000 UTC m=+3221.986782624" Nov 03 23:15:48.406300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:48.406271 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c2abfa-7bef-477b-a881-f3241807a511" path="/var/lib/kubelet/pods/46c2abfa-7bef-477b-a881-f3241807a511/volumes" Nov 03 23:15:49.148785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:49.148745 5051 generic.go:356] "Generic (PLEG): container finished" podID="2e73d6c2-2c6c-4193-8892-a0cc5a38208d" containerID="0e89ad10a630e816a5baa1918eb411366814f889da3cbad162ce628c6920f51d" exitCode=0 Nov 03 23:15:49.149277 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:49.148831 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" event={"ID":"2e73d6c2-2c6c-4193-8892-a0cc5a38208d","Type":"ContainerDied","Data":"0e89ad10a630e816a5baa1918eb411366814f889da3cbad162ce628c6920f51d"} Nov 03 23:15:50.294191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.294168 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:50.329927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.329900 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\") pod \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " Nov 03 23:15:50.330054 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.329943 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\") pod \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " Nov 03 23:15:50.330054 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.329985 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\" (UniqueName: \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\") pod \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\" (UID: \"2e73d6c2-2c6c-4193-8892-a0cc5a38208d\") " Nov 03 23:15:50.330279 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.330255 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca" pod "2e73d6c2-2c6c-4193-8892-a0cc5a38208d" (UID: "2e73d6c2-2c6c-4193-8892-a0cc5a38208d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:50.330369 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.330354 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:50.332248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.332219 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9" pod "2e73d6c2-2c6c-4193-8892-a0cc5a38208d" (UID: "2e73d6c2-2c6c-4193-8892-a0cc5a38208d"). InnerVolumeSpecName "kube-api-access-xh7s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:50.332353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.332286 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca" pod "2e73d6c2-2c6c-4193-8892-a0cc5a38208d" (UID: "2e73d6c2-2c6c-4193-8892-a0cc5a38208d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:50.431076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.431042 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:50.431076 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:50.431066 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7s9\" (UniqueName: \"kubernetes.io/projected/2e73d6c2-2c6c-4193-8892-a0cc5a38208d-kube-api-access-xh7s9\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:15:51.161085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:51.161056 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" Nov 03 23:15:51.161266 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:51.161088 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8" event={"ID":"2e73d6c2-2c6c-4193-8892-a0cc5a38208d","Type":"ContainerDied","Data":"503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29"} Nov 03 23:15:51.161266 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:51.161133 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503dbce33ed9feb28662c91ce61300b0d607e19245cd9b2fdacd79bdc7b72a29" Nov 03 23:15:51.622412 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:15:51.622369 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:15:55.047165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:55.047132 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t"] Nov 03 23:15:55.096726 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:55.096696 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7k65t"] Nov 03 23:15:56.407563 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:15:56.407535 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a66be31-ca52-45ca-9635-bcc58934004f" path="/var/lib/kubelet/pods/8a66be31-ca52-45ca-9635-bcc58934004f/volumes" Nov 03 23:16:02.750197 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:02.750153 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4bjss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:16:02.750654 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:02.750224 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4bjss" podUID="c799271b-6fb6-4e67-9e4a-45f296bf84db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:16:07.048773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:07.048736 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll"] Nov 03 23:16:07.098183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:07.098156 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6wlll"] Nov 03 23:16:08.407462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:08.407429 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241c5c20-47ad-40f4-a780-4c42854782df" path="/var/lib/kubelet/pods/241c5c20-47ad-40f4-a780-4c42854782df/volumes" Nov 03 23:16:12.204742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:12.204708 5051 scope.go:122] "RemoveContainer" containerID="587e304c9be2a3ee043a88f7d2069132348b11f51dca0efc170624becba449af" Nov 03 23:16:19.134746 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.134704 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:16:19.135244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.135181 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2e73d6c2-2c6c-4193-8892-a0cc5a38208d" containerName="revoke-keys" Nov 03 23:16:19.135244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.135195 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e73d6c2-2c6c-4193-8892-a0cc5a38208d" containerName="revoke-keys" Nov 03 23:16:19.135341 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.135285 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e73d6c2-2c6c-4193-8892-a0cc5a38208d" containerName="revoke-keys" Nov 03 23:16:19.139903 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.139887 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.192110 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.192078 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8a134f-openshift-default-dockercfg-8kwfc\"" Nov 03 23:16:19.213493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnhg\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213584 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213662 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213683 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213727 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213773 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213795 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.213876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.214130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.213929 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.248862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.248826 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:16:19.314968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.314928 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.314977 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315035 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315091 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315117 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnhg\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315196 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315256 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315285 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315328 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315403 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315488 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315456 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315622 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.315783 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.315641 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.316084 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.316062 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.317600 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.317577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.317781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.317761 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.376038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.375975 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.376290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.376266 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnhg\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\") pod \"e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.456179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.456101 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:19.889590 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.889558 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:16:19.890416 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:16:19.890393 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0749ce4_7c5e_400e_b8c4_b261bed10e93.slice/crio-b73cf813ea7ca27e438542450a31a5dbceea6871ca3d055882cf0dc31aae75a1 WatchSource:0}: Error finding container b73cf813ea7ca27e438542450a31a5dbceea6871ca3d055882cf0dc31aae75a1: Status 404 returned error can't find the container with id b73cf813ea7ca27e438542450a31a5dbceea6871ca3d055882cf0dc31aae75a1 Nov 03 23:16:19.892683 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.892646 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:19.892803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.892727 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:19.892803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:19.892771 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:20.346997 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:20.346895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerStarted","Data":"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72"} Nov 03 23:16:20.347382 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:20.346979 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerStarted","Data":"b73cf813ea7ca27e438542450a31a5dbceea6871ca3d055882cf0dc31aae75a1"} Nov 03 23:16:20.456411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:20.456375 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:20.461712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:20.461686 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:20.468663 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:20.468612 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podStartSLOduration=1.468599446 podStartE2EDuration="1.468599446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:20.46591345 +0000 UTC m=+3254.203257677" watchObservedRunningTime="2026-11-03 23:16:20.468599446 +0000 UTC m=+3254.205943671" Nov 03 23:16:21.352697 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:21.352663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:21.353810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:21.353792 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:21.625477 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:16:21.625374 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:16:27.191751 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.191710 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:16:27.198900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.198867 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.252409 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.252371 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:16:27.296822 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.296793 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.297030 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.296856 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqngm\" (UniqueName: \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.298509 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.298475 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8a134f\"" Nov 03 23:16:27.298573 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.298531 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-18f2a228-dockercfg-pvvx7\"" Nov 03 23:16:27.358599 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.358574 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:16:27.370052 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.370020 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.397707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.397678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.397848 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.397745 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqngm\" (UniqueName: \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.398345 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.398325 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.428295 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.428269 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:16:27.458761 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.458695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqngm\" (UniqueName: \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\") pod \"payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.498627 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.498597 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.498794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.498647 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54pd\" (UniqueName: \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.515617 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.515579 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:27.600922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.600536 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.600922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.600581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54pd\" (UniqueName: \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.601403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.601382 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.661657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.661624 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54pd\" (UniqueName: \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\") pod \"payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.683473 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.683449 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:27.890933 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:27.890907 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:16:27.892856 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:16:27.892829 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6cc022_c14c_4d19_94ce_d4b8a67407cb.slice/crio-cb4dfe89dc41581d89a714aad7691c42d3bfbc3096e594c68e090ec99676bef4 WatchSource:0}: Error finding container cb4dfe89dc41581d89a714aad7691c42d3bfbc3096e594c68e090ec99676bef4: Status 404 returned error can't find the container with id cb4dfe89dc41581d89a714aad7691c42d3bfbc3096e594c68e090ec99676bef4 Nov 03 23:16:28.060079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.060055 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:16:28.062095 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:16:28.062067 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ffa341_df4d_43b6_942f_129895cee751.slice/crio-435ec0b1ac66e9a5ce1ca810a34127d6a85b5ef2d6404696a808b25791dfed1b WatchSource:0}: Error finding container 435ec0b1ac66e9a5ce1ca810a34127d6a85b5ef2d6404696a808b25791dfed1b: Status 404 returned error can't find the container with id 435ec0b1ac66e9a5ce1ca810a34127d6a85b5ef2d6404696a808b25791dfed1b Nov 03 23:16:28.396754 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.396713 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" event={"ID":"a0ffa341-df4d-43b6-942f-129895cee751","Type":"ContainerStarted","Data":"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3"} Nov 03 23:16:28.397262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.396776 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" event={"ID":"a0ffa341-df4d-43b6-942f-129895cee751","Type":"ContainerStarted","Data":"435ec0b1ac66e9a5ce1ca810a34127d6a85b5ef2d6404696a808b25791dfed1b"} Nov 03 23:16:28.397262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.396814 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:28.398510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.398485 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" event={"ID":"6a6cc022-c14c-4d19-94ce-d4b8a67407cb","Type":"ContainerStarted","Data":"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a"} Nov 03 23:16:28.398647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.398524 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" event={"ID":"6a6cc022-c14c-4d19-94ce-d4b8a67407cb","Type":"ContainerStarted","Data":"cb4dfe89dc41581d89a714aad7691c42d3bfbc3096e594c68e090ec99676bef4"} Nov 03 23:16:28.398647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.398602 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:28.514524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.514473 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" podStartSLOduration=1.514458721 podStartE2EDuration="1.514458721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:28.5117229 +0000 UTC m=+3262.249067128" watchObservedRunningTime="2026-11-03 23:16:28.514458721 +0000 UTC m=+3262.251802945" Nov 03 23:16:28.627474 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:28.627424 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" podStartSLOduration=1.627406417 podStartE2EDuration="1.627406417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:28.626391824 +0000 UTC m=+3262.363736048" watchObservedRunningTime="2026-11-03 23:16:28.627406417 +0000 UTC m=+3262.364750646" Nov 03 23:16:39.408853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:39.408822 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:16:39.409307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:39.408925 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:16:43.054899 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:43.054861 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj"] Nov 03 23:16:43.103962 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:43.103932 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fkslj"] Nov 03 23:16:44.407615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:44.407581 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e" path="/var/lib/kubelet/pods/eb57efb6-8a43-4de5-aba4-bc8fe4b19a8e/volumes" Nov 03 23:16:46.531327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.531249 5051 generic.go:356] "Generic (PLEG): container finished" podID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerID="d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72" exitCode=0 Nov 03 23:16:46.531777 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.531322 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerDied","Data":"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72"} Nov 03 23:16:46.531777 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.531657 5051 scope.go:122] "RemoveContainer" containerID="d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72" Nov 03 23:16:46.532815 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.532788 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:46.532876 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.532857 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:46.532918 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:46.532895 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:16:47.538805 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:47.538774 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerStarted","Data":"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29"} Nov 03 23:16:48.459633 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:48.459238 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:48.459633 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:48.459350 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 23:16:48.459633 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:48.459394 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 23:16:49.481013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:49.467520 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 23:16:49.481013 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:49.467587 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 23:16:50.479014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:50.477892 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 23:16:50.479014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:50.477959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 23:16:51.352873 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:51.352826 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:51.473132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:51.473095 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 23:16:51.474290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:51.474259 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 23:16:51.629132 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:16:51.628787 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:16:52.456880 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:52.456842 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 23:16:52.457329 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:52.456915 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 23:16:53.461122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:53.461091 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:53.462038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:53.462017 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:16:54.046587 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:54.046547 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2"] Nov 03 23:16:54.049264 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:54.049237 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-c4sh2"] Nov 03 23:16:54.406386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:16:54.406350 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798494f5-d826-4885-bc82-626964187563" path="/var/lib/kubelet/pods/798494f5-d826-4885-bc82-626964187563/volumes" Nov 03 23:17:07.355898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:07.355861 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:17:07.384117 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:07.384093 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:17:12.233548 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:12.233515 5051 scope.go:122] "RemoveContainer" containerID="3c13634b1d2ddf859b0fc37f611b18c0cff2215cea43e673a86c91fcd0ba3576" Nov 03 23:17:12.242921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:12.242900 5051 scope.go:122] "RemoveContainer" containerID="8e5b3bef03c4cce9a571af27c3d844932ff8437789857fad78fe4ae8b2158ba9" Nov 03 23:17:12.251417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:12.251390 5051 scope.go:122] "RemoveContainer" containerID="89e95db90fec5d3634e94d9faf4d5b14c517588a015f6d15f0eb4f95bea18010" Nov 03 23:17:21.624799 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:17:21.624755 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:17:29.058102 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:29.058055 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64"] Nov 03 23:17:29.107205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:29.107168 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5jv64"] Nov 03 23:17:30.407225 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:30.407188 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029e9988-0739-4d45-992e-691db1494fd4" path="/var/lib/kubelet/pods/029e9988-0739-4d45-992e-691db1494fd4/volumes" Nov 03 23:17:38.455539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.455508 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b"] Nov 03 23:17:38.459182 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.459163 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.517185 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.517157 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b"] Nov 03 23:17:38.536421 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.536395 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbm2z\" (UniqueName: \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.536568 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.536510 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.536656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.536631 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.563104 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.563079 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:38.563222 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.563201 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:17:38.637858 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.637829 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbm2z\" (UniqueName: \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.638015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.637884 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.638015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.637931 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.638567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.638547 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.640520 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.640501 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.698786 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.698755 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbm2z\" (UniqueName: \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\") pod \"maas-api-revoke-sub-1fd6b91b-8pv5b\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:38.773021 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:38.772917 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:39.149742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:39.149716 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b"] Nov 03 23:17:39.152042 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:17:39.152011 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5953bad_4def_4db3_86e8_8e6036e3b575.slice/crio-cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633 WatchSource:0}: Error finding container cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633: Status 404 returned error can't find the container with id cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633 Nov 03 23:17:39.876650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:39.876618 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" event={"ID":"e5953bad-4def-4db3-86e8-8e6036e3b575","Type":"ContainerStarted","Data":"38757fd47ab7cb15e00df35f4cf056b88d1f3041f8e0ca68b2a7268030d9419b"} Nov 03 23:17:39.877122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:39.876673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" event={"ID":"e5953bad-4def-4db3-86e8-8e6036e3b575","Type":"ContainerStarted","Data":"cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633"} Nov 03 23:17:39.994163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:39.994115 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" podStartSLOduration=1.994102223 podStartE2EDuration="1.994102223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:39.990651228 +0000 UTC m=+3333.727995452" watchObservedRunningTime="2026-11-03 23:17:39.994102223 +0000 UTC m=+3333.731446484" Nov 03 23:17:40.884274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:40.884240 5051 generic.go:356] "Generic (PLEG): container finished" podID="e5953bad-4def-4db3-86e8-8e6036e3b575" containerID="38757fd47ab7cb15e00df35f4cf056b88d1f3041f8e0ca68b2a7268030d9419b" exitCode=0 Nov 03 23:17:40.884681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:40.884326 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" event={"ID":"e5953bad-4def-4db3-86e8-8e6036e3b575","Type":"ContainerDied","Data":"38757fd47ab7cb15e00df35f4cf056b88d1f3041f8e0ca68b2a7268030d9419b"} Nov 03 23:17:42.019294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.019263 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:42.174676 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.174586 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\") pod \"e5953bad-4def-4db3-86e8-8e6036e3b575\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " Nov 03 23:17:42.174855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.174695 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\") pod \"e5953bad-4def-4db3-86e8-8e6036e3b575\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " Nov 03 23:17:42.174855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.174742 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\" (UniqueName: \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\") pod \"e5953bad-4def-4db3-86e8-8e6036e3b575\" (UID: \"e5953bad-4def-4db3-86e8-8e6036e3b575\") " Nov 03 23:17:42.175211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.175184 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca" pod "e5953bad-4def-4db3-86e8-8e6036e3b575" (UID: "e5953bad-4def-4db3-86e8-8e6036e3b575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:17:42.177610 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.177585 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z" pod "e5953bad-4def-4db3-86e8-8e6036e3b575" (UID: "e5953bad-4def-4db3-86e8-8e6036e3b575"). InnerVolumeSpecName "kube-api-access-tbm2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:17:42.177732 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.177682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca" pod "e5953bad-4def-4db3-86e8-8e6036e3b575" (UID: "e5953bad-4def-4db3-86e8-8e6036e3b575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:17:42.276365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.276329 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbm2z\" (UniqueName: \"kubernetes.io/projected/e5953bad-4def-4db3-86e8-8e6036e3b575-kube-api-access-tbm2z\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:42.276365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.276360 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:42.276365 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.276370 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5953bad-4def-4db3-86e8-8e6036e3b575-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:42.896353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.896318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" event={"ID":"e5953bad-4def-4db3-86e8-8e6036e3b575","Type":"ContainerDied","Data":"cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633"} Nov 03 23:17:42.896353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.896355 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b" Nov 03 23:17:42.896574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:42.896364 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6d401290a2fc09f3841bd026ed2c6f41ec1a2e02436d48b47c7eace67f9633" Nov 03 23:17:50.617673 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.617632 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9"] Nov 03 23:17:50.618152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.618136 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e5953bad-4def-4db3-86e8-8e6036e3b575" containerName="revoke-keys" Nov 03 23:17:50.618201 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.618154 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5953bad-4def-4db3-86e8-8e6036e3b575" containerName="revoke-keys" Nov 03 23:17:50.618276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.618266 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5953bad-4def-4db3-86e8-8e6036e3b575" containerName="revoke-keys" Nov 03 23:17:50.621654 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.621638 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.680165 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.680132 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9"] Nov 03 23:17:50.725344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.725312 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:17:50.725585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.725565 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:17:50.753679 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.753652 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.753819 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.753792 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f766\" (UniqueName: \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.753869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.753849 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.854493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.854463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.854623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.854507 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.854623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.854587 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f766\" (UniqueName: \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.855253 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.855228 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.857031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.857010 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.915644 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.915587 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f766\" (UniqueName: \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:50.934462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:50.934442 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:51.308723 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:51.308700 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9"] Nov 03 23:17:51.310873 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:17:51.310846 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615eff2c_ed21_4f0b_8efa_ec6dffbdb216.slice/crio-a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2 WatchSource:0}: Error finding container a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2: Status 404 returned error can't find the container with id a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2 Nov 03 23:17:51.623313 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:17:51.623280 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:17:51.949758 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:51.949720 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" event={"ID":"615eff2c-ed21-4f0b-8efa-ec6dffbdb216","Type":"ContainerStarted","Data":"203a72f7f8decdcd6192c51de80d5be7d0fb12a43d8610ea3532bf7494e961cc"} Nov 03 23:17:51.949975 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:51.949807 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" event={"ID":"615eff2c-ed21-4f0b-8efa-ec6dffbdb216","Type":"ContainerStarted","Data":"a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2"} Nov 03 23:17:52.066759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:52.066687 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" podStartSLOduration=2.066670251 podStartE2EDuration="2.066670251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:52.064562853 +0000 UTC m=+3345.801907077" watchObservedRunningTime="2026-11-03 23:17:52.066670251 +0000 UTC m=+3345.804014477" Nov 03 23:17:52.957051 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:52.956984 5051 generic.go:356] "Generic (PLEG): container finished" podID="615eff2c-ed21-4f0b-8efa-ec6dffbdb216" containerID="203a72f7f8decdcd6192c51de80d5be7d0fb12a43d8610ea3532bf7494e961cc" exitCode=0 Nov 03 23:17:52.957501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:52.957097 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" event={"ID":"615eff2c-ed21-4f0b-8efa-ec6dffbdb216","Type":"ContainerDied","Data":"203a72f7f8decdcd6192c51de80d5be7d0fb12a43d8610ea3532bf7494e961cc"} Nov 03 23:17:54.091248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.091226 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:54.186281 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.186253 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\" (UniqueName: \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\") pod \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " Nov 03 23:17:54.186446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.186346 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\") pod \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " Nov 03 23:17:54.186446 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.186367 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\") pod \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\" (UID: \"615eff2c-ed21-4f0b-8efa-ec6dffbdb216\") " Nov 03 23:17:54.186788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.186754 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca" pod "615eff2c-ed21-4f0b-8efa-ec6dffbdb216" (UID: "615eff2c-ed21-4f0b-8efa-ec6dffbdb216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:17:54.188648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.188622 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca" pod "615eff2c-ed21-4f0b-8efa-ec6dffbdb216" (UID: "615eff2c-ed21-4f0b-8efa-ec6dffbdb216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:17:54.188730 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.188659 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766" pod "615eff2c-ed21-4f0b-8efa-ec6dffbdb216" (UID: "615eff2c-ed21-4f0b-8efa-ec6dffbdb216"). InnerVolumeSpecName "kube-api-access-5f766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:17:54.287618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.287541 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f766\" (UniqueName: \"kubernetes.io/projected/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-kube-api-access-5f766\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:54.287618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.287570 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:54.287618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.287582 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615eff2c-ed21-4f0b-8efa-ec6dffbdb216-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:17:54.969038 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.968971 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" Nov 03 23:17:54.969195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.969006 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9" event={"ID":"615eff2c-ed21-4f0b-8efa-ec6dffbdb216","Type":"ContainerDied","Data":"a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2"} Nov 03 23:17:54.969195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:54.969128 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27341dcd682d0b9d84b98c6624ef888c6a30912538ac2bb55c5fd4b16b4f6a2" Nov 03 23:17:57.058564 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:57.058527 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl"] Nov 03 23:17:57.105905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:57.105873 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7d4jl"] Nov 03 23:17:58.408491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:17:58.408456 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0355cab2-bb9e-4f16-a7d6-00aa15912c93" path="/var/lib/kubelet/pods/0355cab2-bb9e-4f16-a7d6-00aa15912c93/volumes" Nov 03 23:18:02.147528 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:02.147495 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:18:02.255327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:02.255295 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:18:07.148226 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:07.148154 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" containerID="cri-o://69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a" gracePeriod=25 Nov 03 23:18:07.256136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:07.256044 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" containerID="cri-o://09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3" gracePeriod=25 Nov 03 23:18:08.116637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:08.116605 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9"] Nov 03 23:18:08.166051 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:08.166020 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a134f-9jkm9"] Nov 03 23:18:08.405916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:08.405844 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615eff2c-ed21-4f0b-8efa-ec6dffbdb216" path="/var/lib/kubelet/pods/615eff2c-ed21-4f0b-8efa-ec6dffbdb216/volumes" Nov 03 23:18:09.406199 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:09.406157 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:18:09.406690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:09.406207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:18:12.284116 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:12.284085 5051 scope.go:122] "RemoveContainer" containerID="ac60c7343f17f671ab645daa295412e5f1b6522a62fe74deaeaff234d589a415" Nov 03 23:18:12.293748 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:12.293726 5051 scope.go:122] "RemoveContainer" containerID="89cc80aeb436cd3434f67ab7f75486daa03814af115f99f37af51626f296b916" Nov 03 23:18:14.835512 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:14.835481 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:18:14.835978 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:14.835682 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" containerID="cri-o://c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29" gracePeriod=30 Nov 03 23:18:16.409596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.409515 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8"] Nov 03 23:18:16.459533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.459506 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-44zx8"] Nov 03 23:18:16.531299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.531264 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp"] Nov 03 23:18:16.531917 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.531900 5051 cpu_manager.go:414] "containerMap: removing container" podUID="615eff2c-ed21-4f0b-8efa-ec6dffbdb216" containerName="revoke-keys" Nov 03 23:18:16.531963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.531920 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615eff2c-ed21-4f0b-8efa-ec6dffbdb216" containerName="revoke-keys" Nov 03 23:18:16.532079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.532067 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="615eff2c-ed21-4f0b-8efa-ec6dffbdb216" containerName="revoke-keys" Nov 03 23:18:16.537305 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.537285 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.589762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.589739 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:18:16.590261 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.590233 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:18:16.593962 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.593935 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp"] Nov 03 23:18:16.731785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.731703 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzft\" (UniqueName: \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.731785 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.731744 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.731979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.731855 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.832458 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.832430 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzft\" (UniqueName: \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.832458 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.832462 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.832706 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.832530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.833237 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.833211 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.835167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.835144 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:16.893160 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:16.893133 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzft\" (UniqueName: \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\") pod \"maas-api-revoke-sub-3d4156f1-kg8tp\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:17.150170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:17.150145 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:17.526932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:17.526899 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp"] Nov 03 23:18:17.528138 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:18:17.528113 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2e1dcb_a751_4f0f_bc26_85fd458983d3.slice/crio-ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582 WatchSource:0}: Error finding container ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582: Status 404 returned error can't find the container with id ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582 Nov 03 23:18:18.104107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:18.104077 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" event={"ID":"9f2e1dcb-a751-4f0f-bc26-85fd458983d3","Type":"ContainerStarted","Data":"3fd686093917df4c183d0d446c4bb6a5bc4a2cdd7405cd70532a055aed4046c6"} Nov 03 23:18:18.104304 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:18.104125 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" event={"ID":"9f2e1dcb-a751-4f0f-bc26-85fd458983d3","Type":"ContainerStarted","Data":"ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582"} Nov 03 23:18:18.221214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:18.221115 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" podStartSLOduration=2.22109895 podStartE2EDuration="2.22109895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:18:18.218553279 +0000 UTC m=+3371.955897506" watchObservedRunningTime="2026-11-03 23:18:18.22109895 +0000 UTC m=+3371.958443177" Nov 03 23:18:18.407490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:18.407456 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e73d6c2-2c6c-4193-8892-a0cc5a38208d" path="/var/lib/kubelet/pods/2e73d6c2-2c6c-4193-8892-a0cc5a38208d/volumes" Nov 03 23:18:19.110967 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:19.110935 5051 generic.go:356] "Generic (PLEG): container finished" podID="9f2e1dcb-a751-4f0f-bc26-85fd458983d3" containerID="3fd686093917df4c183d0d446c4bb6a5bc4a2cdd7405cd70532a055aed4046c6" exitCode=0 Nov 03 23:18:19.111383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:19.111023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" event={"ID":"9f2e1dcb-a751-4f0f-bc26-85fd458983d3","Type":"ContainerDied","Data":"3fd686093917df4c183d0d446c4bb6a5bc4a2cdd7405cd70532a055aed4046c6"} Nov 03 23:18:19.406423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:19.406338 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:18:19.406423 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:19.406406 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:18:20.089837 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.089798 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:18:20.117983 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.117953 5051 generic.go:356] "Generic (PLEG): container finished" podID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerID="c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29" exitCode=0 Nov 03 23:18:20.118385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.118026 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" Nov 03 23:18:20.118385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.118043 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerDied","Data":"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29"} Nov 03 23:18:20.118385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.118111 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq" event={"ID":"e0749ce4-7c5e-400e-b8c4-b261bed10e93","Type":"ContainerDied","Data":"b73cf813ea7ca27e438542450a31a5dbceea6871ca3d055882cf0dc31aae75a1"} Nov 03 23:18:20.118385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.118136 5051 scope.go:122] "RemoveContainer" containerID="c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29" Nov 03 23:18:20.129195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.129164 5051 scope.go:122] "RemoveContainer" containerID="d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72" Nov 03 23:18:20.142156 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.142130 5051 scope.go:122] "RemoveContainer" containerID="c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29" Nov 03 23:18:20.142547 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:20.142514 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29\": container with ID starting with c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29 not found: ID does not exist" containerID="c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29" Nov 03 23:18:20.142659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.142554 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29"} err="failed to get container status \"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29\": rpc error: code = NotFound desc = could not find container \"c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29\": container with ID starting with c491ceacf2c206970577ce3e25410c25b4ec76f7ef8024c024f1c12363da2e29 not found: ID does not exist" Nov 03 23:18:20.142659 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.142573 5051 scope.go:122] "RemoveContainer" containerID="d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72" Nov 03 23:18:20.142900 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:20.142873 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72\": container with ID starting with d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72 not found: ID does not exist" containerID="d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72" Nov 03 23:18:20.142971 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.142913 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72"} err="failed to get container status \"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72\": rpc error: code = NotFound desc = could not find container \"d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72\": container with ID starting with d7defd1702e0f769bb8c7c76579349d9c1f54375ce3e31703b3ec47598e4aa72 not found: ID does not exist" Nov 03 23:18:20.164419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164391 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164443 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164484 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164523 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164565 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164650 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164686 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164722 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164952 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164772 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\") pod \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\" (UID: \"e0749ce4-7c5e-400e-b8c4-b261bed10e93\") " Nov 03 23:18:20.164952 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164781 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:20.165085 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.164977 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:20.165151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165111 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:20.165235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165216 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.165300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165220 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:20.165300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165242 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.165300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165256 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.165497 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.165478 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:20.167441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.167415 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:20.167564 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.167486 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:20.167639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.167593 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:18:20.168697 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.168677 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg" pod "e0749ce4-7c5e-400e-b8c4-b261bed10e93" (UID: "e0749ce4-7c5e-400e-b8c4-b261bed10e93"). InnerVolumeSpecName "kube-api-access-7wnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:20.262853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.262827 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:20.265799 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265777 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.265888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265802 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.265888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265814 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0749ce4-7c5e-400e-b8c4-b261bed10e93-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.265888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265822 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.265888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265833 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wnhg\" (UniqueName: \"kubernetes.io/projected/e0749ce4-7c5e-400e-b8c4-b261bed10e93-kube-api-access-7wnhg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.265888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.265841 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0749ce4-7c5e-400e-b8c4-b261bed10e93-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.366327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.366295 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\") pod \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " Nov 03 23:18:20.366492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.366441 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\" (UniqueName: \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\") pod \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " Nov 03 23:18:20.366536 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.366508 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\") pod \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\" (UID: \"9f2e1dcb-a751-4f0f-bc26-85fd458983d3\") " Nov 03 23:18:20.366641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.366619 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca" pod "9f2e1dcb-a751-4f0f-bc26-85fd458983d3" (UID: "9f2e1dcb-a751-4f0f-bc26-85fd458983d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:20.366802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.366783 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.368747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.368690 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca" pod "9f2e1dcb-a751-4f0f-bc26-85fd458983d3" (UID: "9f2e1dcb-a751-4f0f-bc26-85fd458983d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:20.368841 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.368777 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft" pod "9f2e1dcb-a751-4f0f-bc26-85fd458983d3" (UID: "9f2e1dcb-a751-4f0f-bc26-85fd458983d3"). InnerVolumeSpecName "kube-api-access-2fzft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:20.468145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.468109 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzft\" (UniqueName: \"kubernetes.io/projected/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-kube-api-access-2fzft\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.468145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.468142 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2e1dcb-a751-4f0f-bc26-85fd458983d3-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:20.544794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.544765 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:18:20.550503 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:20.550477 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a134f-openshift-default-6cfbcbcbc7jr9zq"] Nov 03 23:18:21.126283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:21.126211 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" Nov 03 23:18:21.126662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:21.126210 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp" event={"ID":"9f2e1dcb-a751-4f0f-bc26-85fd458983d3","Type":"ContainerDied","Data":"ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582"} Nov 03 23:18:21.126662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:21.126339 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb3d1059cf0c3e139fd9086d163a5ec61dc3dad134e73d216ab4dd03ddd4582" Nov 03 23:18:21.621598 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:21.621559 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:18:22.406706 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:22.406675 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" path="/var/lib/kubelet/pods/e0749ce4-7c5e-400e-b8c4-b261bed10e93/volumes" Nov 03 23:18:27.058393 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.058362 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm"] Nov 03 23:18:27.108129 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.108102 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6tqpm"] Nov 03 23:18:27.407599 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.407566 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:18:27.434329 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.434294 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\") pod \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " Nov 03 23:18:27.434493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.434406 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\" (UniqueName: \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\") pod \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\" (UID: \"6a6cc022-c14c-4d19-94ce-d4b8a67407cb\") " Nov 03 23:18:27.434742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.434714 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume" pod "6a6cc022-c14c-4d19-94ce-d4b8a67407cb" (UID: "6a6cc022-c14c-4d19-94ce-d4b8a67407cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:27.435605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.435377 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:27.439158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.439130 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm" pod "6a6cc022-c14c-4d19-94ce-d4b8a67407cb" (UID: "6a6cc022-c14c-4d19-94ce-d4b8a67407cb"). InnerVolumeSpecName "kube-api-access-vqngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:27.498493 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.498469 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:18:27.535899 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.535877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\" (UniqueName: \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\") pod \"a0ffa341-df4d-43b6-942f-129895cee751\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " Nov 03 23:18:27.536102 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.535985 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\") pod \"a0ffa341-df4d-43b6-942f-129895cee751\" (UID: \"a0ffa341-df4d-43b6-942f-129895cee751\") " Nov 03 23:18:27.536297 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.536276 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume" pod "a0ffa341-df4d-43b6-942f-129895cee751" (UID: "a0ffa341-df4d-43b6-942f-129895cee751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:27.536445 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.536433 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqngm\" (UniqueName: \"kubernetes.io/projected/6a6cc022-c14c-4d19-94ce-d4b8a67407cb-kube-api-access-vqngm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:27.536489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.536448 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ffa341-df4d-43b6-942f-129895cee751-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:27.538145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.538123 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd" pod "a0ffa341-df4d-43b6-942f-129895cee751" (UID: "a0ffa341-df4d-43b6-942f-129895cee751"). InnerVolumeSpecName "kube-api-access-r54pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:27.636975 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:27.636941 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54pd\" (UniqueName: \"kubernetes.io/projected/a0ffa341-df4d-43b6-942f-129895cee751-kube-api-access-r54pd\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:28.166371 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.166337 5051 generic.go:356] "Generic (PLEG): container finished" podID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerID="69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a" exitCode=0 Nov 03 23:18:28.166776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.166408 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" Nov 03 23:18:28.166776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.166423 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" event={"ID":"6a6cc022-c14c-4d19-94ce-d4b8a67407cb","Type":"ContainerDied","Data":"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a"} Nov 03 23:18:28.166776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.166508 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf" event={"ID":"6a6cc022-c14c-4d19-94ce-d4b8a67407cb","Type":"ContainerDied","Data":"cb4dfe89dc41581d89a714aad7691c42d3bfbc3096e594c68e090ec99676bef4"} Nov 03 23:18:28.166776 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.166536 5051 scope.go:122] "RemoveContainer" containerID="69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a" Nov 03 23:18:28.169256 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.169227 5051 generic.go:356] "Generic (PLEG): container finished" podID="a0ffa341-df4d-43b6-942f-129895cee751" containerID="09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3" exitCode=0 Nov 03 23:18:28.169401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.169283 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" event={"ID":"a0ffa341-df4d-43b6-942f-129895cee751","Type":"ContainerDied","Data":"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3"} Nov 03 23:18:28.169401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.169320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" event={"ID":"a0ffa341-df4d-43b6-942f-129895cee751","Type":"ContainerDied","Data":"435ec0b1ac66e9a5ce1ca810a34127d6a85b5ef2d6404696a808b25791dfed1b"} Nov 03 23:18:28.169401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.169329 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4" Nov 03 23:18:28.177660 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.177643 5051 scope.go:122] "RemoveContainer" containerID="69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a" Nov 03 23:18:28.177952 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:28.177935 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a\": container with ID starting with 69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a not found: ID does not exist" containerID="69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a" Nov 03 23:18:28.178049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.177959 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a"} err="failed to get container status \"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a\": rpc error: code = NotFound desc = could not find container \"69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a\": container with ID starting with 69d70e70e26645bf98fa96b7741eea34a3d002e4074a236e74e2e8994bd6421a not found: ID does not exist" Nov 03 23:18:28.178049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.177973 5051 scope.go:122] "RemoveContainer" containerID="09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3" Nov 03 23:18:28.186458 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.186436 5051 scope.go:122] "RemoveContainer" containerID="09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3" Nov 03 23:18:28.186704 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:28.186686 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3\": container with ID starting with 09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3 not found: ID does not exist" containerID="09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3" Nov 03 23:18:28.186752 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.186712 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3"} err="failed to get container status \"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3\": rpc error: code = NotFound desc = could not find container \"09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3\": container with ID starting with 09cddeee32b70b345ae5ad74e48965a1416f304de68bc8c505a6c90b28d74bc3 not found: ID does not exist" Nov 03 23:18:28.330620 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.330589 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:18:28.380241 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.380215 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a134f-77cfb5fcfnt7zf"] Nov 03 23:18:28.407380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.407354 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d980a15-615c-443c-a6ac-e19233a00217" path="/var/lib/kubelet/pods/3d980a15-615c-443c-a6ac-e19233a00217/volumes" Nov 03 23:18:28.407724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.407712 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" path="/var/lib/kubelet/pods/6a6cc022-c14c-4d19-94ce-d4b8a67407cb/volumes" Nov 03 23:18:28.443321 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.443262 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:18:28.447242 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:28.447220 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a134f-57db7wbxl4"] Nov 03 23:18:30.408088 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:30.408054 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ffa341-df4d-43b6-942f-129895cee751" path="/var/lib/kubelet/pods/a0ffa341-df4d-43b6-942f-129895cee751/volumes" Nov 03 23:18:50.972893 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:50.972859 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 23:18:50.973494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:50.973079 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" podUID="107cdd6b-efe1-4d9e-8646-ed386019fea2" containerName="manager" containerID="cri-o://f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062" gracePeriod=10 Nov 03 23:18:51.047196 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.047164 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs"] Nov 03 23:18:51.076688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.076655 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-wlwhs"] Nov 03 23:18:51.221622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.221601 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 23:18:51.306363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.306294 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\" (UniqueName: \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\") pod \"107cdd6b-efe1-4d9e-8646-ed386019fea2\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " Nov 03 23:18:51.306363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.306326 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\") pod \"107cdd6b-efe1-4d9e-8646-ed386019fea2\" (UID: \"107cdd6b-efe1-4d9e-8646-ed386019fea2\") " Nov 03 23:18:51.306780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.306761 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume" pod "107cdd6b-efe1-4d9e-8646-ed386019fea2" (UID: "107cdd6b-efe1-4d9e-8646-ed386019fea2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:51.308618 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.308597 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c" pod "107cdd6b-efe1-4d9e-8646-ed386019fea2" (UID: "107cdd6b-efe1-4d9e-8646-ed386019fea2"). InnerVolumeSpecName "kube-api-access-jgd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:51.336120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.336096 5051 generic.go:356] "Generic (PLEG): container finished" podID="107cdd6b-efe1-4d9e-8646-ed386019fea2" containerID="f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062" exitCode=0 Nov 03 23:18:51.336235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.336139 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" Nov 03 23:18:51.336235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.336177 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" event={"ID":"107cdd6b-efe1-4d9e-8646-ed386019fea2","Type":"ContainerDied","Data":"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062"} Nov 03 23:18:51.336235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.336229 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd" event={"ID":"107cdd6b-efe1-4d9e-8646-ed386019fea2","Type":"ContainerDied","Data":"391edae764f8905bf72556f066f855ed7ecf6175f01e8247d2a4803de170a7c0"} Nov 03 23:18:51.336362 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.336251 5051 scope.go:122] "RemoveContainer" containerID="f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062" Nov 03 23:18:51.348924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.348903 5051 scope.go:122] "RemoveContainer" containerID="f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062" Nov 03 23:18:51.349308 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:51.349280 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062\": container with ID starting with f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062 not found: ID does not exist" containerID="f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062" Nov 03 23:18:51.349308 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.349318 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062"} err="failed to get container status \"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062\": rpc error: code = NotFound desc = could not find container \"f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062\": container with ID starting with f257f4914383772057469b7854dad3bc624cedec479b07662dbd5687a67b6062 not found: ID does not exist" Nov 03 23:18:51.407221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.407190 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgd7c\" (UniqueName: \"kubernetes.io/projected/107cdd6b-efe1-4d9e-8646-ed386019fea2-kube-api-access-jgd7c\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:51.407221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.407215 5051 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/107cdd6b-efe1-4d9e-8646-ed386019fea2-extensions-socket-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:51.503865 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.503834 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 23:18:51.507325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:51.507296 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pzscd"] Nov 03 23:18:51.622470 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:18:51.622438 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:18:52.405645 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:52.405613 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107cdd6b-efe1-4d9e-8646-ed386019fea2" path="/var/lib/kubelet/pods/107cdd6b-efe1-4d9e-8646-ed386019fea2/volumes" Nov 03 23:18:52.406078 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:18:52.406046 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3" path="/var/lib/kubelet/pods/a73b0704-4e5b-4e4b-81e7-f28b9c21f8e3/volumes" Nov 03 23:19:12.320435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:12.320404 5051 scope.go:122] "RemoveContainer" containerID="cd6469817146060d4f4a3a9368515310ecc7901c2122cba2a9481f4c53c19174" Nov 03 23:19:12.329344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:12.329324 5051 scope.go:122] "RemoveContainer" containerID="c5f005f7df55a447a8105cd834e26bae7e7de5b4e45c8985878079647a1a75c9" Nov 03 23:19:21.621773 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:19:21.621736 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:19:51.622387 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:19:51.622310 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:19:54.573775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.573739 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f"] Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574141 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574153 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574166 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574171 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574184 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" Nov 03 23:19:54.574198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574188 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574205 5051 cpu_manager.go:414] "containerMap: removing container" podUID="107cdd6b-efe1-4d9e-8646-ed386019fea2" containerName="manager" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574210 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107cdd6b-efe1-4d9e-8646-ed386019fea2" containerName="manager" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574225 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9f2e1dcb-a751-4f0f-bc26-85fd458983d3" containerName="revoke-keys" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574230 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2e1dcb-a751-4f0f-bc26-85fd458983d3" containerName="revoke-keys" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574294 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2e1dcb-a751-4f0f-bc26-85fd458983d3" containerName="revoke-keys" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574305 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574311 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6cc022-c14c-4d19-94ce-d4b8a67407cb" containerName="payload-processing" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574319 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ffa341-df4d-43b6-942f-129895cee751" containerName="payload-pre-processing" Nov 03 23:19:54.574406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.574326 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="107cdd6b-efe1-4d9e-8646-ed386019fea2" containerName="manager" Nov 03 23:19:54.577425 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.577411 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.629494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.629455 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f"] Nov 03 23:19:54.676125 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.676086 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6t2gp\"" Nov 03 23:19:54.683793 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.683769 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74946b97-2ba2-465d-ac92-aea0b2b87a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.683918 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.683873 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2jr\" (UniqueName: \"kubernetes.io/projected/74946b97-2ba2-465d-ac92-aea0b2b87a14-kube-api-access-vt2jr\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.785262 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.785235 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74946b97-2ba2-465d-ac92-aea0b2b87a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.785431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.785332 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2jr\" (UniqueName: \"kubernetes.io/projected/74946b97-2ba2-465d-ac92-aea0b2b87a14-kube-api-access-vt2jr\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.785614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.785592 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74946b97-2ba2-465d-ac92-aea0b2b87a14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.846347 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.846286 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2jr\" (UniqueName: \"kubernetes.io/projected/74946b97-2ba2-465d-ac92-aea0b2b87a14-kube-api-access-vt2jr\") pod \"kuadrant-operator-controller-manager-678d87766f-bxz7f\" (UID: \"74946b97-2ba2-465d-ac92-aea0b2b87a14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:54.890196 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:54.890167 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:55.369046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:55.368958 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f"] Nov 03 23:19:55.371921 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:19:55.371894 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74946b97_2ba2_465d_ac92_aea0b2b87a14.slice/crio-eca114f44d1ea956d551cf53bd33c734cef35d97c6b73e6819e0eaea310a4012 WatchSource:0}: Error finding container eca114f44d1ea956d551cf53bd33c734cef35d97c6b73e6819e0eaea310a4012: Status 404 returned error can't find the container with id eca114f44d1ea956d551cf53bd33c734cef35d97c6b73e6819e0eaea310a4012 Nov 03 23:19:55.686015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:55.685905 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" event={"ID":"74946b97-2ba2-465d-ac92-aea0b2b87a14","Type":"ContainerStarted","Data":"67eed7930223cd2bf830842f343d423d826d52697b6dbe5add533c7831362698"} Nov 03 23:19:55.686015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:55.685958 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" event={"ID":"74946b97-2ba2-465d-ac92-aea0b2b87a14","Type":"ContainerStarted","Data":"eca114f44d1ea956d551cf53bd33c734cef35d97c6b73e6819e0eaea310a4012"} Nov 03 23:19:55.686560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:55.686032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:19:55.804817 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:19:55.804755 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" podStartSLOduration=1.804741581 podStartE2EDuration="1.804741581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:19:55.803631874 +0000 UTC m=+3469.540976184" watchObservedRunningTime="2026-11-03 23:19:55.804741581 +0000 UTC m=+3469.542085809" Nov 03 23:20:06.694189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:06.694161 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bxz7f" Nov 03 23:20:12.373363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:12.373331 5051 scope.go:122] "RemoveContainer" containerID="78cd6425f35fe74bf2a8caa38c340b1ee711eff61cfc7d51e216b923ca7a15ab" Nov 03 23:20:12.382643 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:12.382622 5051 scope.go:122] "RemoveContainer" containerID="09e0041d52072ce34e50e76a20bee7fd39607ba70f31c6e2674b1d2244aa126c" Nov 03 23:20:12.390787 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:12.390762 5051 scope.go:122] "RemoveContainer" containerID="28629d67da5a920dff6ccfc957b6a3e19c8acd377218c9e372597b06a1a3263f" Nov 03 23:20:21.622155 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:20:21.622118 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:20:22.609891 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.609857 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg"] Nov 03 23:20:22.610300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.610286 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:20:22.610352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.610302 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:20:22.610397 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.610388 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0749ce4-7c5e-400e-b8c4-b261bed10e93" containerName="istio-proxy" Nov 03 23:20:22.614794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.614776 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.663359 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.663331 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8qx\" (UniqueName: \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.663762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.663452 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.663762 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.663509 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.671418 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.671390 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg"] Nov 03 23:20:22.717191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.717161 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:20:22.717191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.717165 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:20:22.765122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.765090 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.765314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.765186 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8qx\" (UniqueName: \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.765314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.765275 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.766058 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.766034 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.767832 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.767804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.826403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.826370 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8qx\" (UniqueName: \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\") pod \"maas-api-revoke-sub-d576e68b-6bpgg\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:22.928057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:22.927957 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:23.046857 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.046822 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f"] Nov 03 23:20:23.049794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.049769 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6s6f"] Nov 03 23:20:23.302465 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.302438 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg"] Nov 03 23:20:23.305114 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:20:23.305086 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27507f21_7c0d_45de_8eec_f722be5a7f6a.slice/crio-9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6 WatchSource:0}: Error finding container 9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6: Status 404 returned error can't find the container with id 9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6 Nov 03 23:20:23.851979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.851948 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" event={"ID":"27507f21-7c0d-45de-8eec-f722be5a7f6a","Type":"ContainerStarted","Data":"9e1a5b335ae9a5ed8163d89c51c5c1ed0ebd213f12d85a2ea29a844a7c1fb42c"} Nov 03 23:20:23.852438 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.852044 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" event={"ID":"27507f21-7c0d-45de-8eec-f722be5a7f6a","Type":"ContainerStarted","Data":"9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6"} Nov 03 23:20:23.968396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:23.968352 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" podStartSLOduration=1.968339438 podStartE2EDuration="1.968339438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:23.966049083 +0000 UTC m=+3497.703393310" watchObservedRunningTime="2026-11-03 23:20:23.968339438 +0000 UTC m=+3497.705683663" Nov 03 23:20:24.406482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:24.406409 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e4c097-9112-4771-9100-153be774d9ba" path="/var/lib/kubelet/pods/88e4c097-9112-4771-9100-153be774d9ba/volumes" Nov 03 23:20:24.858939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:24.858904 5051 generic.go:356] "Generic (PLEG): container finished" podID="27507f21-7c0d-45de-8eec-f722be5a7f6a" containerID="9e1a5b335ae9a5ed8163d89c51c5c1ed0ebd213f12d85a2ea29a844a7c1fb42c" exitCode=0 Nov 03 23:20:24.859382 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:24.859001 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" event={"ID":"27507f21-7c0d-45de-8eec-f722be5a7f6a","Type":"ContainerDied","Data":"9e1a5b335ae9a5ed8163d89c51c5c1ed0ebd213f12d85a2ea29a844a7c1fb42c"} Nov 03 23:20:26.006769 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.006746 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:26.098885 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.098855 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\") pod \"27507f21-7c0d-45de-8eec-f722be5a7f6a\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " Nov 03 23:20:26.099077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.098907 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\") pod \"27507f21-7c0d-45de-8eec-f722be5a7f6a\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " Nov 03 23:20:26.099077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.099030 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\" (UniqueName: \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\") pod \"27507f21-7c0d-45de-8eec-f722be5a7f6a\" (UID: \"27507f21-7c0d-45de-8eec-f722be5a7f6a\") " Nov 03 23:20:26.099332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.099308 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca" pod "27507f21-7c0d-45de-8eec-f722be5a7f6a" (UID: "27507f21-7c0d-45de-8eec-f722be5a7f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:20:26.101115 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.101096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca" pod "27507f21-7c0d-45de-8eec-f722be5a7f6a" (UID: "27507f21-7c0d-45de-8eec-f722be5a7f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:26.101332 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.101315 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx" pod "27507f21-7c0d-45de-8eec-f722be5a7f6a" (UID: "27507f21-7c0d-45de-8eec-f722be5a7f6a"). InnerVolumeSpecName "kube-api-access-2k8qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:20:26.200132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.200061 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8qx\" (UniqueName: \"kubernetes.io/projected/27507f21-7c0d-45de-8eec-f722be5a7f6a-kube-api-access-2k8qx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:26.200132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.200090 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:26.200132 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.200105 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27507f21-7c0d-45de-8eec-f722be5a7f6a-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:26.871346 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.871314 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" Nov 03 23:20:26.871523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.871355 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg" event={"ID":"27507f21-7c0d-45de-8eec-f722be5a7f6a","Type":"ContainerDied","Data":"9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6"} Nov 03 23:20:26.871523 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:26.871404 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1a08b07cbf195541412414119976a3f572856be82e6b709089d1edc43bb8a6" Nov 03 23:20:34.047788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:34.047756 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s"] Nov 03 23:20:34.097443 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:34.097405 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c2b2s"] Nov 03 23:20:34.407922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:34.407894 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a99749-e059-488e-9132-ae97a9de7596" path="/var/lib/kubelet/pods/13a99749-e059-488e-9132-ae97a9de7596/volumes" Nov 03 23:20:51.558029 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.557977 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs"] Nov 03 23:20:51.558435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.558410 5051 cpu_manager.go:414] "containerMap: removing container" podUID="27507f21-7c0d-45de-8eec-f722be5a7f6a" containerName="revoke-keys" Nov 03 23:20:51.558435 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.558420 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27507f21-7c0d-45de-8eec-f722be5a7f6a" containerName="revoke-keys" Nov 03 23:20:51.558527 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.558496 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="27507f21-7c0d-45de-8eec-f722be5a7f6a" containerName="revoke-keys" Nov 03 23:20:51.561570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.561554 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.616122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.616081 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:20:51.628352 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:20:51.628310 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:20:51.631863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.631831 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs"] Nov 03 23:20:51.640567 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.640540 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttnt\" (UniqueName: \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.640709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.640614 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.640709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.640681 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.665242 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.665223 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:20:51.741692 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.741669 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.741820 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.741727 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttnt\" (UniqueName: \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.741820 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.741768 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.742327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.742297 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.744147 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.744124 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.802226 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.802204 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttnt\" (UniqueName: \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\") pod \"maas-api-revoke-sub-d2bc93bc-qxghs\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:51.874552 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:51.874528 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:52.250156 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:52.250101 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs"] Nov 03 23:20:52.253006 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:20:52.252964 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d70b6cf_b723_4e28_9e24_bb013f59b646.slice/crio-95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4 WatchSource:0}: Error finding container 95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4: Status 404 returned error can't find the container with id 95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4 Nov 03 23:20:53.020972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:53.020939 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" event={"ID":"7d70b6cf-b723-4e28-9e24-bb013f59b646","Type":"ContainerStarted","Data":"1e3b1c226903c1e92aeae02f6ad012e5965c96fef65a46539a4f4a6327f7f6ae"} Nov 03 23:20:53.021400 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:53.021013 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" event={"ID":"7d70b6cf-b723-4e28-9e24-bb013f59b646","Type":"ContainerStarted","Data":"95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4"} Nov 03 23:20:53.136979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:53.136927 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" podStartSLOduration=2.136909428 podStartE2EDuration="2.136909428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:53.135350899 +0000 UTC m=+3526.872695124" watchObservedRunningTime="2026-11-03 23:20:53.136909428 +0000 UTC m=+3526.874253654" Nov 03 23:20:54.027397 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:54.027361 5051 generic.go:356] "Generic (PLEG): container finished" podID="7d70b6cf-b723-4e28-9e24-bb013f59b646" containerID="1e3b1c226903c1e92aeae02f6ad012e5965c96fef65a46539a4f4a6327f7f6ae" exitCode=0 Nov 03 23:20:54.028048 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:54.027445 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" event={"ID":"7d70b6cf-b723-4e28-9e24-bb013f59b646","Type":"ContainerDied","Data":"1e3b1c226903c1e92aeae02f6ad012e5965c96fef65a46539a4f4a6327f7f6ae"} Nov 03 23:20:55.168787 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.168760 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:20:55.273982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.273954 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\") pod \"7d70b6cf-b723-4e28-9e24-bb013f59b646\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " Nov 03 23:20:55.274120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.274057 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\") pod \"7d70b6cf-b723-4e28-9e24-bb013f59b646\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " Nov 03 23:20:55.274120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.274107 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\" (UniqueName: \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\") pod \"7d70b6cf-b723-4e28-9e24-bb013f59b646\" (UID: \"7d70b6cf-b723-4e28-9e24-bb013f59b646\") " Nov 03 23:20:55.274405 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.274382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca" pod "7d70b6cf-b723-4e28-9e24-bb013f59b646" (UID: "7d70b6cf-b723-4e28-9e24-bb013f59b646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:20:55.276168 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.276140 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca" pod "7d70b6cf-b723-4e28-9e24-bb013f59b646" (UID: "7d70b6cf-b723-4e28-9e24-bb013f59b646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:55.276260 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.276207 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt" pod "7d70b6cf-b723-4e28-9e24-bb013f59b646" (UID: "7d70b6cf-b723-4e28-9e24-bb013f59b646"). InnerVolumeSpecName "kube-api-access-2ttnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:20:55.375747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.375720 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:55.375747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.375747 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttnt\" (UniqueName: \"kubernetes.io/projected/7d70b6cf-b723-4e28-9e24-bb013f59b646-kube-api-access-2ttnt\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:55.375915 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:55.375757 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d70b6cf-b723-4e28-9e24-bb013f59b646-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:20:56.040145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:56.040104 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" event={"ID":"7d70b6cf-b723-4e28-9e24-bb013f59b646","Type":"ContainerDied","Data":"95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4"} Nov 03 23:20:56.040325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:56.040157 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95379fa1894824f561b72fb94738eec2ebdb524c88dd79974a0a6b5e814e8ae4" Nov 03 23:20:56.040325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:20:56.040121 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs" Nov 03 23:21:04.370191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.370159 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:21:04.370709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.370692 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7d70b6cf-b723-4e28-9e24-bb013f59b646" containerName="revoke-keys" Nov 03 23:21:04.370775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.370712 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d70b6cf-b723-4e28-9e24-bb013f59b646" containerName="revoke-keys" Nov 03 23:21:04.370823 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.370814 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d70b6cf-b723-4e28-9e24-bb013f59b646" containerName="revoke-keys" Nov 03 23:21:04.374152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.374135 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.427623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.427583 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-60ef36-openshift-default-dockercfg-5spcc\"" Nov 03 23:21:04.454836 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.454787 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.454836 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.454834 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26v2t\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.454961 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455020 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455115 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455152 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455231 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.455370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.455270 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.484917 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.484889 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:21:04.556693 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556710 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556734 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556759 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556781 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556821 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556849 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.556928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556922 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.556944 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26v2t\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557348 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.557151 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.557360 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.557413 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557580 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.557490 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.557637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.557581 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.559730 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.559707 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.560148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.560129 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.617564 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.617530 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.617742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.617718 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26v2t\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\") pod \"e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:04.687148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:04.687119 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:05.135752 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:05.135704 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:21:05.138313 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:21:05.138280 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882395fd_3812_417c_8eaa_d30861dde0bc.slice/crio-3bca3e750d3e3c275aae0caf01ae6c00dcaf5c7b9adeb80160d2dee6c560a801 WatchSource:0}: Error finding container 3bca3e750d3e3c275aae0caf01ae6c00dcaf5c7b9adeb80160d2dee6c560a801: Status 404 returned error can't find the container with id 3bca3e750d3e3c275aae0caf01ae6c00dcaf5c7b9adeb80160d2dee6c560a801 Nov 03 23:21:05.140352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:05.140318 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:05.140431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:05.140384 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:05.140431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:05.140411 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:06.101247 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:06.101207 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerStarted","Data":"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778"} Nov 03 23:21:06.101775 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:06.101266 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerStarted","Data":"3bca3e750d3e3c275aae0caf01ae6c00dcaf5c7b9adeb80160d2dee6c560a801"} Nov 03 23:21:06.243819 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:06.243764 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podStartSLOduration=2.243748483 podStartE2EDuration="2.243748483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:06.241150582 +0000 UTC m=+3539.978494808" watchObservedRunningTime="2026-11-03 23:21:06.243748483 +0000 UTC m=+3539.981092748" Nov 03 23:21:06.687317 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:06.687283 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:06.692314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:06.692291 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:07.108684 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:07.108647 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:07.109745 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:07.109722 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:12.423715 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:12.423676 5051 scope.go:122] "RemoveContainer" containerID="33d14634965bcc07e2f7973f8b1e10d9f81d47b90c3eaddf4d0aff0728e8a1b9" Nov 03 23:21:12.975429 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:12.975393 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:21:12.982145 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:12.982091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.032202 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.032170 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:21:13.078092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.078051 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-13aad129-dockercfg-shvj2\"" Nov 03 23:21:13.078092 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.078051 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-60ef36\"" Nov 03 23:21:13.136275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.136248 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:21:13.138297 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.138264 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.138470 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.138431 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jkf\" (UniqueName: \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.140039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.140016 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.197662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.197633 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:21:13.239866 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.239793 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.240046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.239873 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czgr\" (UniqueName: \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.240046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.239897 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.240046 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.239939 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jkf\" (UniqueName: \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.240487 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.240466 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.300597 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.300568 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jkf\" (UniqueName: \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\") pod \"payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.341703 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.341667 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czgr\" (UniqueName: \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.341909 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.341713 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.342464 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.342443 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.402409 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.402381 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czgr\" (UniqueName: \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\") pod \"payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.452900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.452867 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:13.598172 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.598144 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:13.831881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.831831 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:21:13.834364 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:21:13.834342 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fe9db3_acbb_43ad_b69e_55901d3f1555.slice/crio-6c6615a847d7c2a69dd85d870f25fde3bb9ec4e6f63957f599cf4fc56a6e194b WatchSource:0}: Error finding container 6c6615a847d7c2a69dd85d870f25fde3bb9ec4e6f63957f599cf4fc56a6e194b: Status 404 returned error can't find the container with id 6c6615a847d7c2a69dd85d870f25fde3bb9ec4e6f63957f599cf4fc56a6e194b Nov 03 23:21:13.974258 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:13.974233 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:21:13.976351 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:21:13.976323 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3b21de_2a9b_42f4_a4f2_8c3392cfacd5.slice/crio-a587d90bbdbd70abd02b6f5f50245c56a9210fcc73688f7be000127db252c63f WatchSource:0}: Error finding container a587d90bbdbd70abd02b6f5f50245c56a9210fcc73688f7be000127db252c63f: Status 404 returned error can't find the container with id a587d90bbdbd70abd02b6f5f50245c56a9210fcc73688f7be000127db252c63f Nov 03 23:21:14.145920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.145879 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" event={"ID":"e5fe9db3-acbb-43ad-b69e-55901d3f1555","Type":"ContainerStarted","Data":"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4"} Nov 03 23:21:14.146098 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.145979 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" event={"ID":"e5fe9db3-acbb-43ad-b69e-55901d3f1555","Type":"ContainerStarted","Data":"6c6615a847d7c2a69dd85d870f25fde3bb9ec4e6f63957f599cf4fc56a6e194b"} Nov 03 23:21:14.146158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.146110 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:14.147581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.147554 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" event={"ID":"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5","Type":"ContainerStarted","Data":"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9"} Nov 03 23:21:14.147691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.147597 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" event={"ID":"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5","Type":"ContainerStarted","Data":"a587d90bbdbd70abd02b6f5f50245c56a9210fcc73688f7be000127db252c63f"} Nov 03 23:21:14.147691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.147634 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:14.262397 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.262350 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" podStartSLOduration=1.262337594 podStartE2EDuration="1.262337594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:14.260198488 +0000 UTC m=+3547.997542713" watchObservedRunningTime="2026-11-03 23:21:14.262337594 +0000 UTC m=+3547.999681818" Nov 03 23:21:14.376879 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:14.376807 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" podStartSLOduration=2.37678454 podStartE2EDuration="2.37678454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:14.374493495 +0000 UTC m=+3548.111837730" watchObservedRunningTime="2026-11-03 23:21:14.37678454 +0000 UTC m=+3548.114128783" Nov 03 23:21:21.622803 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:21:21.622723 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:21:25.155691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:25.155663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:21:25.156275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:25.155732 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:21:33.255724 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.255693 5051 generic.go:356] "Generic (PLEG): container finished" podID="882395fd-3812-417c-8eaa-d30861dde0bc" containerID="f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778" exitCode=0 Nov 03 23:21:33.256251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.255734 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerDied","Data":"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778"} Nov 03 23:21:33.256251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.256059 5051 scope.go:122] "RemoveContainer" containerID="f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778" Nov 03 23:21:33.257215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.257176 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:33.257315 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.257252 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:33.257315 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:33.257292 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:21:34.279017 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:34.268285 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerStarted","Data":"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b"} Nov 03 23:21:34.688339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:34.688304 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:34.689901 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:34.689868 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 23:21:34.690091 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:34.689944 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 23:21:35.688419 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:35.688376 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 23:21:35.688920 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:35.688450 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 23:21:36.688478 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:36.688443 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 23:21:36.688924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:36.688502 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 23:21:37.108846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:37.108823 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:37.688022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:37.687968 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 23:21:37.688217 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:37.688046 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 23:21:38.687968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:38.687926 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 23:21:38.688482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:38.688025 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 23:21:39.691921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:39.691880 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:39.692950 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:21:39.692926 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:21:51.622575 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:21:51.622534 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:22:07.397223 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:07.397194 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:22:07.429818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:07.429794 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:22:12.446130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:12.446098 5051 scope.go:122] "RemoveContainer" containerID="0e89ad10a630e816a5baa1918eb411366814f889da3cbad162ce628c6920f51d" Nov 03 23:22:12.457077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:12.457058 5051 scope.go:122] "RemoveContainer" containerID="2bffcac1be1545a7ba9fc6cbe0973a45a62b71856df2394f9d2eab653b0ad55c" Nov 03 23:22:12.465450 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:12.465430 5051 scope.go:122] "RemoveContainer" containerID="3d3d75f8c7ecec476150016cb046c26a7b918f6d95effc3eba6cde0203eda2c7" Nov 03 23:22:21.624877 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:22:21.624838 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:22:22.701462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.701422 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs"] Nov 03 23:22:22.706739 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.706716 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.762322 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.762278 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.762322 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.762321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.762586 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.762358 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9m5l\" (UniqueName: \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.762960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.762934 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs"] Nov 03 23:22:22.808902 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.808875 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:22:22.808902 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.808874 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:22:22.863033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.863001 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9m5l\" (UniqueName: \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.863170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.863121 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.863170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.863145 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.863727 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.863706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.865710 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.865691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:22.923387 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:22.923355 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9m5l\" (UniqueName: \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\") pod \"maas-api-revoke-sub-4892429a-zk9fs\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:23.020185 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:23.020117 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:23.399246 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:23.399215 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs"] Nov 03 23:22:23.399732 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:22:23.399709 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed8942_51ce_4a61_a5a5_7f73db286281.slice/crio-754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3 WatchSource:0}: Error finding container 754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3: Status 404 returned error can't find the container with id 754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3 Nov 03 23:22:23.552875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:23.552845 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" event={"ID":"dbed8942-51ce-4a61-a5a5-7f73db286281","Type":"ContainerStarted","Data":"754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3"} Nov 03 23:22:24.559976 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:24.559942 5051 generic.go:356] "Generic (PLEG): container finished" podID="dbed8942-51ce-4a61-a5a5-7f73db286281" containerID="6249fd56ce0860bcf2a67b02afafa17e592fcec0e93b3a889b08a813b7f842f4" exitCode=0 Nov 03 23:22:24.560481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:24.560026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" event={"ID":"dbed8942-51ce-4a61-a5a5-7f73db286281","Type":"ContainerDied","Data":"6249fd56ce0860bcf2a67b02afafa17e592fcec0e93b3a889b08a813b7f842f4"} Nov 03 23:22:25.708062 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.708036 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:25.793260 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.793225 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\") pod \"dbed8942-51ce-4a61-a5a5-7f73db286281\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " Nov 03 23:22:25.793406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.793277 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\" (UniqueName: \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\") pod \"dbed8942-51ce-4a61-a5a5-7f73db286281\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " Nov 03 23:22:25.793406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.793394 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\") pod \"dbed8942-51ce-4a61-a5a5-7f73db286281\" (UID: \"dbed8942-51ce-4a61-a5a5-7f73db286281\") " Nov 03 23:22:25.793747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.793728 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca" pod "dbed8942-51ce-4a61-a5a5-7f73db286281" (UID: "dbed8942-51ce-4a61-a5a5-7f73db286281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:25.793849 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.793832 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:25.795609 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.795586 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca" pod "dbed8942-51ce-4a61-a5a5-7f73db286281" (UID: "dbed8942-51ce-4a61-a5a5-7f73db286281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:25.795717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.795635 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l" pod "dbed8942-51ce-4a61-a5a5-7f73db286281" (UID: "dbed8942-51ce-4a61-a5a5-7f73db286281"). InnerVolumeSpecName "kube-api-access-g9m5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:25.895347 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.895303 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbed8942-51ce-4a61-a5a5-7f73db286281-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:25.895347 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:25.895346 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9m5l\" (UniqueName: \"kubernetes.io/projected/dbed8942-51ce-4a61-a5a5-7f73db286281-kube-api-access-g9m5l\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:26.573221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:26.573197 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" Nov 03 23:22:26.573396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:26.573190 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs" event={"ID":"dbed8942-51ce-4a61-a5a5-7f73db286281","Type":"ContainerDied","Data":"754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3"} Nov 03 23:22:26.573396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:26.573320 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754b435edf6c568dff74c89a0053f3e36dda045d519f10d17cc0d7f866c92fc3" Nov 03 23:22:34.258023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.254476 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9"] Nov 03 23:22:34.258023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.255616 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dbed8942-51ce-4a61-a5a5-7f73db286281" containerName="revoke-keys" Nov 03 23:22:34.258023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.255637 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbed8942-51ce-4a61-a5a5-7f73db286281" containerName="revoke-keys" Nov 03 23:22:34.258023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.255881 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbed8942-51ce-4a61-a5a5-7f73db286281" containerName="revoke-keys" Nov 03 23:22:34.262275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.262248 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.314804 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.314778 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9"] Nov 03 23:22:34.317412 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.317388 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:22:34.360979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.360947 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:22:34.380518 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.380485 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5dv\" (UniqueName: \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.380661 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.380540 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.380748 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.380724 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.481482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.481443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.481653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.481515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.481653 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.481580 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5dv\" (UniqueName: \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.482162 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.482140 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.483960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.483938 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.542088 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.542027 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5dv\" (UniqueName: \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.575484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.575460 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:34.952472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:34.952345 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9"] Nov 03 23:22:34.955234 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:22:34.955202 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ab6db_5802_4a15_a856_47446989fbe8.slice/crio-a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f WatchSource:0}: Error finding container a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f: Status 404 returned error can't find the container with id a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f Nov 03 23:22:35.621292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:35.621259 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" event={"ID":"fc4ab6db-5802-4a15-a856-47446989fbe8","Type":"ContainerStarted","Data":"023a75f9184008b4dc163ddaa7547c2a15db6d86943f9653ac254a0c081899cd"} Nov 03 23:22:35.621722 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:35.621312 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" event={"ID":"fc4ab6db-5802-4a15-a856-47446989fbe8","Type":"ContainerStarted","Data":"a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f"} Nov 03 23:22:35.736719 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:35.736665 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" podStartSLOduration=1.7366517240000001 podStartE2EDuration="1.736651724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:22:35.735597197 +0000 UTC m=+3629.472941423" watchObservedRunningTime="2026-11-03 23:22:35.736651724 +0000 UTC m=+3629.473995949" Nov 03 23:22:36.629731 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:36.629698 5051 generic.go:356] "Generic (PLEG): container finished" podID="fc4ab6db-5802-4a15-a856-47446989fbe8" containerID="023a75f9184008b4dc163ddaa7547c2a15db6d86943f9653ac254a0c081899cd" exitCode=0 Nov 03 23:22:36.630177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:36.629787 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" event={"ID":"fc4ab6db-5802-4a15-a856-47446989fbe8","Type":"ContainerDied","Data":"023a75f9184008b4dc163ddaa7547c2a15db6d86943f9653ac254a0c081899cd"} Nov 03 23:22:37.781058 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.781037 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:37.917496 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.917408 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\") pod \"fc4ab6db-5802-4a15-a856-47446989fbe8\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " Nov 03 23:22:37.917707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.917521 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\") pod \"fc4ab6db-5802-4a15-a856-47446989fbe8\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " Nov 03 23:22:37.917707 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.917627 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\" (UniqueName: \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\") pod \"fc4ab6db-5802-4a15-a856-47446989fbe8\" (UID: \"fc4ab6db-5802-4a15-a856-47446989fbe8\") " Nov 03 23:22:37.917952 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.917912 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca" pod "fc4ab6db-5802-4a15-a856-47446989fbe8" (UID: "fc4ab6db-5802-4a15-a856-47446989fbe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:37.918126 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.918106 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:37.920140 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.920118 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca" pod "fc4ab6db-5802-4a15-a856-47446989fbe8" (UID: "fc4ab6db-5802-4a15-a856-47446989fbe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:37.922112 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:37.922092 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv" pod "fc4ab6db-5802-4a15-a856-47446989fbe8" (UID: "fc4ab6db-5802-4a15-a856-47446989fbe8"). InnerVolumeSpecName "kube-api-access-4h5dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:38.026921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:38.020528 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4ab6db-5802-4a15-a856-47446989fbe8-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:38.026921 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:38.020572 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5dv\" (UniqueName: \"kubernetes.io/projected/fc4ab6db-5802-4a15-a856-47446989fbe8-kube-api-access-4h5dv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:38.643927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:38.643895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" event={"ID":"fc4ab6db-5802-4a15-a856-47446989fbe8","Type":"ContainerDied","Data":"a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f"} Nov 03 23:22:38.643927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:38.643930 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9" Nov 03 23:22:38.644183 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:38.643944 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a760679ab3d0903df42d4be0761ea3a9d5c1d680d14046536b3f37e08d54dc0f" Nov 03 23:22:43.047095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:43.047060 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b"] Nov 03 23:22:43.097821 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:43.097787 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd6b91b-8pv5b"] Nov 03 23:22:44.406517 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:44.406483 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5953bad-4def-4db3-86e8-8e6036e3b575" path="/var/lib/kubelet/pods/e5953bad-4def-4db3-86e8-8e6036e3b575/volumes" Nov 03 23:22:45.734909 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:45.734825 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:22:45.841059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:45.841026 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:22:50.735315 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:50.735251 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" podUID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" containerName="payload-processing" containerID="cri-o://7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9" gracePeriod=25 Nov 03 23:22:50.841313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:50.841264 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" podUID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" containerName="payload-pre-processing" containerID="cri-o://1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4" gracePeriod=25 Nov 03 23:22:50.993855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:50.993787 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:22:51.062556 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.062520 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\" (UniqueName: \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\") pod \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " Nov 03 23:22:51.062772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.062690 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\") pod \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\" (UID: \"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5\") " Nov 03 23:22:51.063098 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.063064 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume" pod "0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" (UID: "0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:51.063251 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.063229 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:51.065592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.065565 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf" pod "0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" (UID: "0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5"). InnerVolumeSpecName "kube-api-access-t4jkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:51.089662 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.089639 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:22:51.164118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.164088 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\") pod \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " Nov 03 23:22:51.164314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.164200 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\" (UniqueName: \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\") pod \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\" (UID: \"e5fe9db3-acbb-43ad-b69e-55901d3f1555\") " Nov 03 23:22:51.164410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.164396 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jkf\" (UniqueName: \"kubernetes.io/projected/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5-kube-api-access-t4jkf\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:51.164464 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.164444 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume" pod "e5fe9db3-acbb-43ad-b69e-55901d3f1555" (UID: "e5fe9db3-acbb-43ad-b69e-55901d3f1555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:51.166429 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.166409 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr" pod "e5fe9db3-acbb-43ad-b69e-55901d3f1555" (UID: "e5fe9db3-acbb-43ad-b69e-55901d3f1555"). InnerVolumeSpecName "kube-api-access-7czgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:51.266163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.266081 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fe9db3-acbb-43ad-b69e-55901d3f1555-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:51.266163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.266112 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czgr\" (UniqueName: \"kubernetes.io/projected/e5fe9db3-acbb-43ad-b69e-55901d3f1555-kube-api-access-7czgr\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:51.622548 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:22:51.622516 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:22:51.716535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.716503 5051 generic.go:356] "Generic (PLEG): container finished" podID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" containerID="7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9" exitCode=0 Nov 03 23:22:51.716749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.716558 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" Nov 03 23:22:51.716749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.716594 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" event={"ID":"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5","Type":"ContainerDied","Data":"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9"} Nov 03 23:22:51.716749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.716649 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7" event={"ID":"0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5","Type":"ContainerDied","Data":"a587d90bbdbd70abd02b6f5f50245c56a9210fcc73688f7be000127db252c63f"} Nov 03 23:22:51.716749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.716675 5051 scope.go:122] "RemoveContainer" containerID="7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9" Nov 03 23:22:51.718126 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.718104 5051 generic.go:356] "Generic (PLEG): container finished" podID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" containerID="1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4" exitCode=0 Nov 03 23:22:51.718243 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.718153 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" Nov 03 23:22:51.718243 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.718153 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" event={"ID":"e5fe9db3-acbb-43ad-b69e-55901d3f1555","Type":"ContainerDied","Data":"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4"} Nov 03 23:22:51.718243 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.718210 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8" event={"ID":"e5fe9db3-acbb-43ad-b69e-55901d3f1555","Type":"ContainerDied","Data":"6c6615a847d7c2a69dd85d870f25fde3bb9ec4e6f63957f599cf4fc56a6e194b"} Nov 03 23:22:51.726896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.726876 5051 scope.go:122] "RemoveContainer" containerID="7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9" Nov 03 23:22:51.727196 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:22:51.727173 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9\": container with ID starting with 7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9 not found: ID does not exist" containerID="7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9" Nov 03 23:22:51.727316 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.727199 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9"} err="failed to get container status \"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9\": rpc error: code = NotFound desc = could not find container \"7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9\": container with ID starting with 7e215570a0911b051c568cb6e5065a41f0aef1fd0ef815f0ee878e1a8af3b7e9 not found: ID does not exist" Nov 03 23:22:51.727316 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.727214 5051 scope.go:122] "RemoveContainer" containerID="1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4" Nov 03 23:22:51.735652 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.735636 5051 scope.go:122] "RemoveContainer" containerID="1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4" Nov 03 23:22:51.735904 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:22:51.735886 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4\": container with ID starting with 1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4 not found: ID does not exist" containerID="1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4" Nov 03 23:22:51.735946 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.735907 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4"} err="failed to get container status \"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4\": rpc error: code = NotFound desc = could not find container \"1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4\": container with ID starting with 1a473d7ebe9be77d9ed81ec51796f72e2659da28062c0fe7c597bc05d7a1b3f4 not found: ID does not exist" Nov 03 23:22:51.880313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.880234 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:22:51.886584 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.886556 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-60ef36-656858f446mbq7"] Nov 03 23:22:51.995189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.995161 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:22:51.999972 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:51.999946 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-60ef36-77855qtqr8"] Nov 03 23:22:52.406051 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:52.405953 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" path="/var/lib/kubelet/pods/0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5/volumes" Nov 03 23:22:52.406421 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:52.406405 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" path="/var/lib/kubelet/pods/e5fe9db3-acbb-43ad-b69e-55901d3f1555/volumes" Nov 03 23:22:54.108492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:54.108457 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9"] Nov 03 23:22:54.158328 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:54.158300 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-60ef36-8nww9"] Nov 03 23:22:54.407572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:22:54.407499 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4ab6db-5802-4a15-a856-47446989fbe8" path="/var/lib/kubelet/pods/fc4ab6db-5802-4a15-a856-47446989fbe8/volumes" Nov 03 23:23:00.839542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:00.839506 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:23:00.840047 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:00.839767 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" containerID="cri-o://67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b" gracePeriod=30 Nov 03 23:23:06.099960 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.099928 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:23:06.220862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.220828 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.220876 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.220913 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.220958 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221133 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221168 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:06.221283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221229 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221264 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221292 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221289 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221305 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221342 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\") pod \"882395fd-3812-417c-8eaa-d30861dde0bc\" (UID: \"882395fd-3812-417c-8eaa-d30861dde0bc\") " Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221361 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221660 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221763 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221781 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221796 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.221812 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221808 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/882395fd-3812-417c-8eaa-d30861dde0bc-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.222235 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.221824 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.223540 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.223511 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:06.223696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.223680 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "kube-api-access-26v2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:06.224036 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.224020 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:06.224111 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.224047 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo" pod "882395fd-3812-417c-8eaa-d30861dde0bc" (UID: "882395fd-3812-417c-8eaa-d30861dde0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:23:06.322691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.322654 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/882395fd-3812-417c-8eaa-d30861dde0bc-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.322691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.322686 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/882395fd-3812-417c-8eaa-d30861dde0bc-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.322691 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.322699 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.322934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.322711 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26v2t\" (UniqueName: \"kubernetes.io/projected/882395fd-3812-417c-8eaa-d30861dde0bc-kube-api-access-26v2t\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:23:06.816216 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.816183 5051 generic.go:356] "Generic (PLEG): container finished" podID="882395fd-3812-417c-8eaa-d30861dde0bc" containerID="67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b" exitCode=0 Nov 03 23:23:06.816401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.816229 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" Nov 03 23:23:06.816401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.816264 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerDied","Data":"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b"} Nov 03 23:23:06.816401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.816331 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5" event={"ID":"882395fd-3812-417c-8eaa-d30861dde0bc","Type":"ContainerDied","Data":"3bca3e750d3e3c275aae0caf01ae6c00dcaf5c7b9adeb80160d2dee6c560a801"} Nov 03 23:23:06.816401 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.816360 5051 scope.go:122] "RemoveContainer" containerID="67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b" Nov 03 23:23:06.826184 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.826165 5051 scope.go:122] "RemoveContainer" containerID="f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778" Nov 03 23:23:06.834828 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.834801 5051 scope.go:122] "RemoveContainer" containerID="67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b" Nov 03 23:23:06.835110 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:23:06.835089 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b\": container with ID starting with 67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b not found: ID does not exist" containerID="67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b" Nov 03 23:23:06.835166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.835119 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b"} err="failed to get container status \"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b\": rpc error: code = NotFound desc = could not find container \"67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b\": container with ID starting with 67016fde57968246438e3742cabb372c99bc9274df0886431517761eb081158b not found: ID does not exist" Nov 03 23:23:06.835166 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.835137 5051 scope.go:122] "RemoveContainer" containerID="f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778" Nov 03 23:23:06.835366 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:23:06.835349 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778\": container with ID starting with f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778 not found: ID does not exist" containerID="f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778" Nov 03 23:23:06.835410 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.835371 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778"} err="failed to get container status \"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778\": rpc error: code = NotFound desc = could not find container \"f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778\": container with ID starting with f9f2eb28123f1afe531e6451688ceb7a299103fae1fe6e69e7bffc3772169778 not found: ID does not exist" Nov 03 23:23:06.942853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.942818 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:23:06.943434 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943414 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fc4ab6db-5802-4a15-a856-47446989fbe8" containerName="revoke-keys" Nov 03 23:23:06.943434 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943435 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4ab6db-5802-4a15-a856-47446989fbe8" containerName="revoke-keys" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943454 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" containerName="payload-processing" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943459 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" containerName="payload-processing" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943478 5051 cpu_manager.go:414] "containerMap: removing container" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943484 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943496 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" containerName="payload-pre-processing" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943501 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" containerName="payload-pre-processing" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943510 5051 cpu_manager.go:414] "containerMap: removing container" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:23:06.943570 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943515 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:23:06.943854 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943575 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3b21de-2a9b-42f4-a4f2-8c3392cfacd5" containerName="payload-processing" Nov 03 23:23:06.943854 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943587 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:23:06.943854 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943595 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fe9db3-acbb-43ad-b69e-55901d3f1555" containerName="payload-pre-processing" Nov 03 23:23:06.943854 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.943603 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4ab6db-5802-4a15-a856-47446989fbe8" containerName="revoke-keys" Nov 03 23:23:06.947981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:06.947964 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.002585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.002335 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-374e2aad-kserve-self-signed-certs\"" Nov 03 23:23:07.007071 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.007038 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:23:07.028803 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.028774 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:23:07.054082 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.054054 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-60ef36-openshift-default-86dc6c59d5qhpc5"] Nov 03 23:23:07.128895 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.128861 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.129292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.128900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwf67\" (UniqueName: \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.129292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.128922 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.129292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.129054 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.129292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.129127 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.129292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.129174 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230079 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230144 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwf67\" (UniqueName: \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230175 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230205 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230313 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230548 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230352 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230554 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.230755 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.230733 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.232778 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.232750 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.233211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.233189 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.292302 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.292272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwf67\" (UniqueName: \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\") pod \"e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.560170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.560091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:07.935110 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:07.935085 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:23:07.937552 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:23:07.937521 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3063790b_9c44_4fb3_8613_3542b9d4642d.slice/crio-ce5eea038e4014522a9c17189af5340fe64c2689c93fb435c676896844339e2e WatchSource:0}: Error finding container ce5eea038e4014522a9c17189af5340fe64c2689c93fb435c676896844339e2e: Status 404 returned error can't find the container with id ce5eea038e4014522a9c17189af5340fe64c2689c93fb435c676896844339e2e Nov 03 23:23:08.406629 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:08.406588 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" path="/var/lib/kubelet/pods/882395fd-3812-417c-8eaa-d30861dde0bc/volumes" Nov 03 23:23:08.830780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:08.830682 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" event={"ID":"3063790b-9c44-4fb3-8613-3542b9d4642d","Type":"ContainerStarted","Data":"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035"} Nov 03 23:23:08.830780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:08.830725 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" event={"ID":"3063790b-9c44-4fb3-8613-3542b9d4642d","Type":"ContainerStarted","Data":"ce5eea038e4014522a9c17189af5340fe64c2689c93fb435c676896844339e2e"} Nov 03 23:23:08.948477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:08.948416 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" podStartSLOduration=2.9484022 podStartE2EDuration="2.9484022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:23:08.94660884 +0000 UTC m=+3662.683953077" watchObservedRunningTime="2026-11-03 23:23:08.9484022 +0000 UTC m=+3662.685746418" Nov 03 23:23:09.560852 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:09.560816 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:17.560277 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:17.560244 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:17.572977 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:17.572944 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:17.969039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:17.969011 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:23:20.047282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:20.047249 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp"] Nov 03 23:23:20.097403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:20.097366 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kg8tp"] Nov 03 23:23:20.406490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:23:20.406461 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2e1dcb-a751-4f0f-bc26-85fd458983d3" path="/var/lib/kubelet/pods/9f2e1dcb-a751-4f0f-bc26-85fd458983d3/volumes" Nov 03 23:23:21.622965 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:23:21.622928 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:23:51.623429 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:23:51.623379 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:24:12.522763 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:12.522730 5051 scope.go:122] "RemoveContainer" containerID="38757fd47ab7cb15e00df35f4cf056b88d1f3041f8e0ca68b2a7268030d9419b" Nov 03 23:24:12.532298 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:12.532277 5051 scope.go:122] "RemoveContainer" containerID="203a72f7f8decdcd6192c51de80d5be7d0fb12a43d8610ea3532bf7494e961cc" Nov 03 23:24:21.622101 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:24:21.622015 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:24:24.250276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.250241 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj"] Nov 03 23:24:24.250790 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.250725 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="882395fd-3812-417c-8eaa-d30861dde0bc" containerName="istio-proxy" Nov 03 23:24:24.255517 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.255497 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.299789 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.299763 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.299929 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.299909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8sk\" (UniqueName: \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.300040 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.300019 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.357734 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.357704 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:24:24.357916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.357779 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:24:24.360546 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.360525 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj"] Nov 03 23:24:24.400900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.400877 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.401030 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.400929 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.401030 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.401020 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8sk\" (UniqueName: \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.401624 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.401605 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.403504 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.403474 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.462082 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.462052 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8sk\" (UniqueName: \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\") pod \"maas-api-revoke-sub-452b6d7b-qvsfj\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.568837 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.568760 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:24.949339 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:24.949313 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj"] Nov 03 23:24:24.951720 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:24:24.951687 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802f9f2b_9d3f_41e9_9886_fd2406d28465.slice/crio-b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa WatchSource:0}: Error finding container b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa: Status 404 returned error can't find the container with id b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa Nov 03 23:24:25.346058 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:25.346022 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" event={"ID":"802f9f2b-9d3f-41e9-9886-fd2406d28465","Type":"ContainerStarted","Data":"635d655680173cbe7b9011ab3d9c4bca4c9fa6dceeed78028f34dacc3cb07b83"} Nov 03 23:24:25.346498 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:25.346084 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" event={"ID":"802f9f2b-9d3f-41e9-9886-fd2406d28465","Type":"ContainerStarted","Data":"b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa"} Nov 03 23:24:25.461459 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:25.461402 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" podStartSLOduration=1.461387692 podStartE2EDuration="1.461387692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:24:25.460507809 +0000 UTC m=+3739.197852034" watchObservedRunningTime="2026-11-03 23:24:25.461387692 +0000 UTC m=+3739.198731916" Nov 03 23:24:26.353506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:26.353467 5051 generic.go:356] "Generic (PLEG): container finished" podID="802f9f2b-9d3f-41e9-9886-fd2406d28465" containerID="635d655680173cbe7b9011ab3d9c4bca4c9fa6dceeed78028f34dacc3cb07b83" exitCode=0 Nov 03 23:24:26.354064 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:26.353553 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" event={"ID":"802f9f2b-9d3f-41e9-9886-fd2406d28465","Type":"ContainerDied","Data":"635d655680173cbe7b9011ab3d9c4bca4c9fa6dceeed78028f34dacc3cb07b83"} Nov 03 23:24:27.888592 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:27.888300 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:27.946576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:27.946527 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\") pod \"802f9f2b-9d3f-41e9-9886-fd2406d28465\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " Nov 03 23:24:27.946757 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:27.946586 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\") pod \"802f9f2b-9d3f-41e9-9886-fd2406d28465\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " Nov 03 23:24:27.947194 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:27.947165 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca" pod "802f9f2b-9d3f-41e9-9886-fd2406d28465" (UID: "802f9f2b-9d3f-41e9-9886-fd2406d28465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:24:27.948836 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:27.948818 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca" pod "802f9f2b-9d3f-41e9-9886-fd2406d28465" (UID: "802f9f2b-9d3f-41e9-9886-fd2406d28465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:24:28.048014 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.047905 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\" (UniqueName: \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\") pod \"802f9f2b-9d3f-41e9-9886-fd2406d28465\" (UID: \"802f9f2b-9d3f-41e9-9886-fd2406d28465\") " Nov 03 23:24:28.050063 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.048274 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:28.050063 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.048300 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802f9f2b-9d3f-41e9-9886-fd2406d28465-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:28.050906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.050859 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk" pod "802f9f2b-9d3f-41e9-9886-fd2406d28465" (UID: "802f9f2b-9d3f-41e9-9886-fd2406d28465"). InnerVolumeSpecName "kube-api-access-gm8sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:24:28.149420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.149371 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm8sk\" (UniqueName: \"kubernetes.io/projected/802f9f2b-9d3f-41e9-9886-fd2406d28465-kube-api-access-gm8sk\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:28.444696 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.444608 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" Nov 03 23:24:28.445409 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.445380 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj" event={"ID":"802f9f2b-9d3f-41e9-9886-fd2406d28465","Type":"ContainerDied","Data":"b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa"} Nov 03 23:24:28.445604 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:28.445587 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b631a4cdb16b9079de7b419582c5723d6bad963135574320825537851bec8aaa" Nov 03 23:24:37.201715 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:37.201678 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:24:51.623407 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:24:51.623365 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:24:52.222688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.222644 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" podUID="3063790b-9c44-4fb3-8613-3542b9d4642d" containerName="main" containerID="cri-o://1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035" gracePeriod=45 Nov 03 23:24:52.487180 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.487114 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:24:52.525916 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.525889 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526099 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.525972 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\" (UniqueName: \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526099 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526014 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526099 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526084 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526110 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526131 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\") pod \"3063790b-9c44-4fb3-8613-3542b9d4642d\" (UID: \"3063790b-9c44-4fb3-8613-3542b9d4642d\") " Nov 03 23:24:52.526276 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526182 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:24:52.526453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526365 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:24:52.526453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526420 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-tmp-dir\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.526600 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.526450 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:24:52.528494 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.528460 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:24:52.528614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.528499 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "kube-api-access-vwf67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:24:52.528614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.528514 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs" pod "3063790b-9c44-4fb3-8613-3542b9d4642d" (UID: "3063790b-9c44-4fb3-8613-3542b9d4642d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:24:52.592582 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.592545 5051 generic.go:356] "Generic (PLEG): container finished" podID="3063790b-9c44-4fb3-8613-3542b9d4642d" containerID="1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035" exitCode=0 Nov 03 23:24:52.592809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.592601 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" Nov 03 23:24:52.592809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.592625 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" event={"ID":"3063790b-9c44-4fb3-8613-3542b9d4642d","Type":"ContainerDied","Data":"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035"} Nov 03 23:24:52.592809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.592673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px" event={"ID":"3063790b-9c44-4fb3-8613-3542b9d4642d","Type":"ContainerDied","Data":"ce5eea038e4014522a9c17189af5340fe64c2689c93fb435c676896844339e2e"} Nov 03 23:24:52.592809 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.592693 5051 scope.go:122] "RemoveContainer" containerID="1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035" Nov 03 23:24:52.606863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.606831 5051 scope.go:122] "RemoveContainer" containerID="1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035" Nov 03 23:24:52.607261 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:24:52.607236 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035\": container with ID starting with 1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035 not found: ID does not exist" containerID="1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035" Nov 03 23:24:52.607405 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.607269 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035"} err="failed to get container status \"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035\": rpc error: code = NotFound desc = could not find container \"1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035\": container with ID starting with 1b38630a35c8ff72bfb67baf8f4f79bb0257ae61a2047a50f5592fc353324035 not found: ID does not exist" Nov 03 23:24:52.627457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.627429 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwf67\" (UniqueName: \"kubernetes.io/projected/3063790b-9c44-4fb3-8613-3542b9d4642d-kube-api-access-vwf67\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.627457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.627459 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3063790b-9c44-4fb3-8613-3542b9d4642d-tls-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.627957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.627473 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-home\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.627957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.627481 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-dshm\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.627957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.627491 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3063790b-9c44-4fb3-8613-3542b9d4642d-model-cache\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:52.773816 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.773731 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:24:52.777881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:52.777848 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-374e2aad-kserve-859c94668f-xs7px"] Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.905593 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89"] Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906521 5051 cpu_manager.go:414] "containerMap: removing container" podUID="802f9f2b-9d3f-41e9-9886-fd2406d28465" containerName="revoke-keys" Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906556 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802f9f2b-9d3f-41e9-9886-fd2406d28465" containerName="revoke-keys" Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906583 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3063790b-9c44-4fb3-8613-3542b9d4642d" containerName="main" Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906591 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3063790b-9c44-4fb3-8613-3542b9d4642d" containerName="main" Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906736 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="802f9f2b-9d3f-41e9-9886-fd2406d28465" containerName="revoke-keys" Nov 03 23:24:53.911015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.906753 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3063790b-9c44-4fb3-8613-3542b9d4642d" containerName="main" Nov 03 23:24:53.933109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.931769 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:53.951927 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:53.951901 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89"] Nov 03 23:24:54.016041 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.015308 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:24:54.016233 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.016084 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:24:54.044524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.044476 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.044524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.044521 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.044813 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.044571 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7rv\" (UniqueName: \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.145268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.145230 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.145463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.145278 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.145463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.145360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7rv\" (UniqueName: \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.146220 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.146192 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.147877 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.147856 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.236306 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.236221 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7rv\" (UniqueName: \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\") pod \"maas-api-revoke-sub-3e31a506-qxh89\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.250681 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.250651 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:54.435593 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.435552 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3063790b-9c44-4fb3-8613-3542b9d4642d" path="/var/lib/kubelet/pods/3063790b-9c44-4fb3-8613-3542b9d4642d/volumes" Nov 03 23:24:54.836351 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:54.833779 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89"] Nov 03 23:24:55.649344 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:55.649309 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" event={"ID":"e9e33a56-418c-42af-8333-4223d934bef6","Type":"ContainerStarted","Data":"973a9d56eb955126ea717e3212d4f39a4c33e25da16565f199be326c4184ed09"} Nov 03 23:24:55.649874 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:55.649367 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" event={"ID":"e9e33a56-418c-42af-8333-4223d934bef6","Type":"ContainerStarted","Data":"f394bef34f165885943fcc46a9efbf346fcd02cc052403f73a2b2396439b2067"} Nov 03 23:24:55.767522 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:55.767473 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" podStartSLOduration=2.767457505 podStartE2EDuration="2.767457505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:24:55.766815884 +0000 UTC m=+3769.504160112" watchObservedRunningTime="2026-11-03 23:24:55.767457505 +0000 UTC m=+3769.504801730" Nov 03 23:24:56.657177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:56.657142 5051 generic.go:356] "Generic (PLEG): container finished" podID="e9e33a56-418c-42af-8333-4223d934bef6" containerID="973a9d56eb955126ea717e3212d4f39a4c33e25da16565f199be326c4184ed09" exitCode=0 Nov 03 23:24:56.657634 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:56.657241 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" event={"ID":"e9e33a56-418c-42af-8333-4223d934bef6","Type":"ContainerDied","Data":"973a9d56eb955126ea717e3212d4f39a4c33e25da16565f199be326c4184ed09"} Nov 03 23:24:57.801483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.801458 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:57.898822 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.898789 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\") pod \"e9e33a56-418c-42af-8333-4223d934bef6\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " Nov 03 23:24:57.899023 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.898891 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\" (UniqueName: \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\") pod \"e9e33a56-418c-42af-8333-4223d934bef6\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " Nov 03 23:24:57.899077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.899034 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\") pod \"e9e33a56-418c-42af-8333-4223d934bef6\" (UID: \"e9e33a56-418c-42af-8333-4223d934bef6\") " Nov 03 23:24:57.899411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.899382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca" pod "e9e33a56-418c-42af-8333-4223d934bef6" (UID: "e9e33a56-418c-42af-8333-4223d934bef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:24:57.901121 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.901091 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca" pod "e9e33a56-418c-42af-8333-4223d934bef6" (UID: "e9e33a56-418c-42af-8333-4223d934bef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:24:57.901212 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:57.901141 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv" pod "e9e33a56-418c-42af-8333-4223d934bef6" (UID: "e9e33a56-418c-42af-8333-4223d934bef6"). InnerVolumeSpecName "kube-api-access-xc7rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:24:58.000357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.000272 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:58.000357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.000306 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e33a56-418c-42af-8333-4223d934bef6-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:58.000357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.000317 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7rv\" (UniqueName: \"kubernetes.io/projected/e9e33a56-418c-42af-8333-4223d934bef6-kube-api-access-xc7rv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:24:58.670257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.670164 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" event={"ID":"e9e33a56-418c-42af-8333-4223d934bef6","Type":"ContainerDied","Data":"f394bef34f165885943fcc46a9efbf346fcd02cc052403f73a2b2396439b2067"} Nov 03 23:24:58.670257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.670206 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89" Nov 03 23:24:58.670257 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:24:58.670224 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f394bef34f165885943fcc46a9efbf346fcd02cc052403f73a2b2396439b2067" Nov 03 23:25:12.557109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:12.557075 5051 scope.go:122] "RemoveContainer" containerID="3fd686093917df4c183d0d446c4bb6a5bc4a2cdd7405cd70532a055aed4046c6" Nov 03 23:25:21.622764 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:25:21.622727 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:25:23.542770 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.542729 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl"] Nov 03 23:25:23.543384 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.543361 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e9e33a56-418c-42af-8333-4223d934bef6" containerName="revoke-keys" Nov 03 23:25:23.543475 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.543387 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e33a56-418c-42af-8333-4223d934bef6" containerName="revoke-keys" Nov 03 23:25:23.543543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.543532 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e33a56-418c-42af-8333-4223d934bef6" containerName="revoke-keys" Nov 03 23:25:23.548250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.548226 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.600574 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.600529 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:25:23.606373 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.606331 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvq4\" (UniqueName: \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.606533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.606472 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.606533 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.606516 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.649926 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.649899 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 23:25:23.652349 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.652328 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl"] Nov 03 23:25:23.707179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.707150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.707335 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.707215 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvq4\" (UniqueName: \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.707335 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.707266 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.707887 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.707865 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.709668 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.709648 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.767947 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.767920 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvq4\" (UniqueName: \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\") pod \"maas-api-revoke-sub-a7a59534-s8xfl\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:23.861900 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:23.861875 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:24.238312 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:24.238287 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl"] Nov 03 23:25:24.241267 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:25:24.241235 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41807ea_d48c_472c_92cd_6aac85d57e54.slice/crio-92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c WatchSource:0}: Error finding container 92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c: Status 404 returned error can't find the container with id 92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c Nov 03 23:25:24.843109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:24.843018 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" event={"ID":"c41807ea-d48c-472c-92cd-6aac85d57e54","Type":"ContainerStarted","Data":"04253ffeb8756e02336f1252596ac2fa97816d1a2109657fdeb174440178ee1b"} Nov 03 23:25:24.843109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:24.843070 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" event={"ID":"c41807ea-d48c-472c-92cd-6aac85d57e54","Type":"ContainerStarted","Data":"92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c"} Nov 03 23:25:24.960039 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:24.959953 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" podStartSLOduration=1.959938564 podStartE2EDuration="1.959938564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:24.957311946 +0000 UTC m=+3798.694656172" watchObservedRunningTime="2026-11-03 23:25:24.959938564 +0000 UTC m=+3798.697282786" Nov 03 23:25:25.850507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:25.850476 5051 generic.go:356] "Generic (PLEG): container finished" podID="c41807ea-d48c-472c-92cd-6aac85d57e54" containerID="04253ffeb8756e02336f1252596ac2fa97816d1a2109657fdeb174440178ee1b" exitCode=0 Nov 03 23:25:25.850907 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:25.850562 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" event={"ID":"c41807ea-d48c-472c-92cd-6aac85d57e54","Type":"ContainerDied","Data":"04253ffeb8756e02336f1252596ac2fa97816d1a2109657fdeb174440178ee1b"} Nov 03 23:25:26.046067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:26.046033 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg"] Nov 03 23:25:26.048813 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:26.048786 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6bpgg"] Nov 03 23:25:26.419709 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:26.419676 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27507f21-7c0d-45de-8eec-f722be5a7f6a" path="/var/lib/kubelet/pods/27507f21-7c0d-45de-8eec-f722be5a7f6a/volumes" Nov 03 23:25:26.993380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:26.993355 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:27.037393 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.037362 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\") pod \"c41807ea-d48c-472c-92cd-6aac85d57e54\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " Nov 03 23:25:27.037581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.037413 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\") pod \"c41807ea-d48c-472c-92cd-6aac85d57e54\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " Nov 03 23:25:27.037581 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.037448 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\" (UniqueName: \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\") pod \"c41807ea-d48c-472c-92cd-6aac85d57e54\" (UID: \"c41807ea-d48c-472c-92cd-6aac85d57e54\") " Nov 03 23:25:27.037820 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.037788 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca" pod "c41807ea-d48c-472c-92cd-6aac85d57e54" (UID: "c41807ea-d48c-472c-92cd-6aac85d57e54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:25:27.039741 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.039719 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca" pod "c41807ea-d48c-472c-92cd-6aac85d57e54" (UID: "c41807ea-d48c-472c-92cd-6aac85d57e54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:25:27.039832 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.039764 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4" pod "c41807ea-d48c-472c-92cd-6aac85d57e54" (UID: "c41807ea-d48c-472c-92cd-6aac85d57e54"). InnerVolumeSpecName "kube-api-access-zfvq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:25:27.138484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.138402 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:27.138484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.138435 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41807ea-d48c-472c-92cd-6aac85d57e54-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:27.138484 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.138446 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvq4\" (UniqueName: \"kubernetes.io/projected/c41807ea-d48c-472c-92cd-6aac85d57e54-kube-api-access-zfvq4\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:27.861543 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.861512 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" Nov 03 23:25:27.861698 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.861514 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl" event={"ID":"c41807ea-d48c-472c-92cd-6aac85d57e54","Type":"ContainerDied","Data":"92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c"} Nov 03 23:25:27.861698 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:27.861644 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b047e5b321c2bf8d29ba6ae0e2eed542100d61c658a58592b082251a80bf6c" Nov 03 23:25:43.743503 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.743474 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:25:43.743924 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.743912 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c41807ea-d48c-472c-92cd-6aac85d57e54" containerName="revoke-keys" Nov 03 23:25:43.743971 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.743925 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41807ea-d48c-472c-92cd-6aac85d57e54" containerName="revoke-keys" Nov 03 23:25:43.744022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.744012 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41807ea-d48c-472c-92cd-6aac85d57e54" containerName="revoke-keys" Nov 03 23:25:43.747123 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.747107 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.799453 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.799419 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-738d0c9a-openshift-default-dockercfg-z48b2\"" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806507 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806558 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvqj\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806597 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806675 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806856 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806881 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.813011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.806933 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.851645 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.851608 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:25:43.908406 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908363 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908413 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvqj\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908498 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908588 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908712 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908785 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908811 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908824 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.908862 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908859 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.909214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.908940 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.909214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.909150 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.909299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.909210 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.909702 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.909673 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.911077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.911056 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.911788 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.911767 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.969794 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.969768 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:43.969919 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:43.969884 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvqj\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\") pod \"e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:44.060718 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.060658 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:44.489711 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.489686 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:25:44.491638 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:25:44.491609 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e13648_c1f9_41d4_a455_68597db4c4de.slice/crio-82070c232f4f341d266124460a39ab625d5ce311adaac6711f1510c1525b7141 WatchSource:0}: Error finding container 82070c232f4f341d266124460a39ab625d5ce311adaac6711f1510c1525b7141: Status 404 returned error can't find the container with id 82070c232f4f341d266124460a39ab625d5ce311adaac6711f1510c1525b7141 Nov 03 23:25:44.493742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.493710 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:25:44.493826 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.493773 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:25:44.493826 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.493804 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:25:44.956749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.956707 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerStarted","Data":"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410"} Nov 03 23:25:44.957170 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:44.956771 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerStarted","Data":"82070c232f4f341d266124460a39ab625d5ce311adaac6711f1510c1525b7141"} Nov 03 23:25:45.060913 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:45.060879 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:45.066102 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:45.066080 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:45.077797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:45.077755 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podStartSLOduration=2.077740935 podStartE2EDuration="2.077740935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:45.074888552 +0000 UTC m=+3818.812232776" watchObservedRunningTime="2026-11-03 23:25:45.077740935 +0000 UTC m=+3818.815085159" Nov 03 23:25:45.963354 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:45.963322 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:45.964316 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:45.964299 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:25:51.085664 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.085625 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:25:51.090252 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.090227 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.151010 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.150966 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-738d0c9a\"" Nov 03 23:25:51.185827 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.185791 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.185979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.185902 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwk2\" (UniqueName: \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.200694 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.200668 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-738d0c9a-dockercfg-wg6nd\"" Nov 03 23:25:51.203223 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.203201 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:25:51.276327 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.276287 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:25:51.280064 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.280040 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.286792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.286660 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.286792 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.286745 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwk2\" (UniqueName: \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.287385 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.287366 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.352456 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.352430 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:25:51.356922 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.356895 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwk2\" (UniqueName: \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\") pod \"payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.388108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.388069 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.388108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.388111 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvnl\" (UniqueName: \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.409860 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.409835 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:51.488703 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.488568 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.488703 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.488613 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvnl\" (UniqueName: \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.489319 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.489288 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.549603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.549575 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvnl\" (UniqueName: \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\") pod \"payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.593422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.593395 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:51.625778 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:25:51.625736 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:25:51.787289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.787263 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:25:51.789746 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:25:51.789719 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab4a9f1_12fb_4ba4_9260_49ababc545ff.slice/crio-e1feb922b2469d408f37cea0a7ff711185daf4698bd48ec4545e00efe0516406 WatchSource:0}: Error finding container e1feb922b2469d408f37cea0a7ff711185daf4698bd48ec4545e00efe0516406: Status 404 returned error can't find the container with id e1feb922b2469d408f37cea0a7ff711185daf4698bd48ec4545e00efe0516406 Nov 03 23:25:51.973968 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:51.973945 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:25:51.976128 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:25:51.976101 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb633fff_35d9_4607_ab4e_db59cb21e860.slice/crio-8d0440e7f30cf5f7cb6fba89cababd42e0e7d8855374409e5748125c58e05c64 WatchSource:0}: Error finding container 8d0440e7f30cf5f7cb6fba89cababd42e0e7d8855374409e5748125c58e05c64: Status 404 returned error can't find the container with id 8d0440e7f30cf5f7cb6fba89cababd42e0e7d8855374409e5748125c58e05c64 Nov 03 23:25:52.005962 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:52.005932 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" event={"ID":"7ab4a9f1-12fb-4ba4-9260-49ababc545ff","Type":"ContainerStarted","Data":"3dfe9a63bc563cedc7bbab3b15142888063f7d3bb8025d8970d2ff964a73a0c4"} Nov 03 23:25:52.006096 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:52.006073 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:25:52.006167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:52.006115 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" event={"ID":"7ab4a9f1-12fb-4ba4-9260-49ababc545ff","Type":"ContainerStarted","Data":"e1feb922b2469d408f37cea0a7ff711185daf4698bd48ec4545e00efe0516406"} Nov 03 23:25:52.007189 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:52.007167 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" event={"ID":"cb633fff-35d9-4607-ab4e-db59cb21e860","Type":"ContainerStarted","Data":"8d0440e7f30cf5f7cb6fba89cababd42e0e7d8855374409e5748125c58e05c64"} Nov 03 23:25:52.122793 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:52.122733 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" podStartSLOduration=1.122719124 podStartE2EDuration="1.122719124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:52.121737488 +0000 UTC m=+3825.859081714" watchObservedRunningTime="2026-11-03 23:25:52.122719124 +0000 UTC m=+3825.860063343" Nov 03 23:25:53.013877 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:53.013833 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" event={"ID":"cb633fff-35d9-4607-ab4e-db59cb21e860","Type":"ContainerStarted","Data":"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a"} Nov 03 23:25:53.014124 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:53.013975 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:25:53.130094 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:53.130041 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" podStartSLOduration=2.13002887 podStartE2EDuration="2.13002887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:53.127929216 +0000 UTC m=+3826.865273441" watchObservedRunningTime="2026-11-03 23:25:53.13002887 +0000 UTC m=+3826.867373094" Nov 03 23:25:56.045914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:56.045879 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs"] Nov 03 23:25:56.095244 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:56.095217 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qxghs"] Nov 03 23:25:56.407277 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:25:56.407243 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d70b6cf-b723-4e28-9e24-bb013f59b646" path="/var/lib/kubelet/pods/7d70b6cf-b723-4e28-9e24-bb013f59b646/volumes" Nov 03 23:26:03.018221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:03.018187 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:26:04.022591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:04.022564 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:26:11.140683 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.140650 5051 generic.go:356] "Generic (PLEG): container finished" podID="66e13648-c1f9-41d4-a455-68597db4c4de" containerID="879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410" exitCode=0 Nov 03 23:26:11.141187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.140721 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerDied","Data":"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410"} Nov 03 23:26:11.141187 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.141118 5051 scope.go:122] "RemoveContainer" containerID="879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410" Nov 03 23:26:11.142357 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.142327 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:11.142490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.142411 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:11.142490 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:11.142459 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:12.147913 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:12.147880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerStarted","Data":"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87"} Nov 03 23:26:13.075016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.062311 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:13.081152 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.081116 5051 patch_prober.go:29] interesting pod/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Nov 03 23:26:13.081293 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.081182 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Nov 03 23:26:13.681658 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.681622 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn"] Nov 03 23:26:13.685736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.685713 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.717939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.717914 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7qj\" (UniqueName: \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.718095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.717961 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.718095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.717980 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.787481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.787446 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mjvcl\"" Nov 03 23:26:13.787481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.787446 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:26:13.790388 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.790364 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn"] Nov 03 23:26:13.819378 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.819352 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7qj\" (UniqueName: \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.819509 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.819411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.819509 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.819438 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.820064 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.820041 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.822003 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.821965 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.882525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.882499 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7qj\" (UniqueName: \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\") pod \"maas-api-revoke-keys-e2e-ait-738d0c9a-925cn\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:13.999067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:13.998956 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:14.090912 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:14.087425 5051 patch_prober.go:29] interesting pod/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Nov 03 23:26:14.090912 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:14.087485 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Nov 03 23:26:14.382725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:14.382696 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn"] Nov 03 23:26:14.386132 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:26:14.386102 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14caff3e_c495_4c8a_96d3_9b60436a5bda.slice/crio-94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281 WatchSource:0}: Error finding container 94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281: Status 404 returned error can't find the container with id 94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281 Nov 03 23:26:15.070108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.066407 5051 patch_prober.go:29] interesting pod/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Nov 03 23:26:15.070108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.066469 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Nov 03 23:26:15.190338 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.190306 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" event={"ID":"14caff3e-c495-4c8a-96d3-9b60436a5bda","Type":"ContainerStarted","Data":"9a947791e829fade70a0edc890d170d67d3f4fcba5005fccf014a1619e5d85c1"} Nov 03 23:26:15.190524 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.190359 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" event={"ID":"14caff3e-c495-4c8a-96d3-9b60436a5bda","Type":"ContainerStarted","Data":"94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281"} Nov 03 23:26:15.307518 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.307459 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" podStartSLOduration=2.307442567 podStartE2EDuration="2.307442567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:15.306108705 +0000 UTC m=+3849.043452932" watchObservedRunningTime="2026-11-03 23:26:15.307442567 +0000 UTC m=+3849.044786804" Nov 03 23:26:15.978015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:15.970084 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:16.077015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:16.075379 5051 patch_prober.go:29] interesting pod/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Nov 03 23:26:16.077015 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:16.075440 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Nov 03 23:26:16.197283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:16.197249 5051 generic.go:356] "Generic (PLEG): container finished" podID="14caff3e-c495-4c8a-96d3-9b60436a5bda" containerID="9a947791e829fade70a0edc890d170d67d3f4fcba5005fccf014a1619e5d85c1" exitCode=0 Nov 03 23:26:16.197482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:16.197334 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" event={"ID":"14caff3e-c495-4c8a-96d3-9b60436a5bda","Type":"ContainerDied","Data":"9a947791e829fade70a0edc890d170d67d3f4fcba5005fccf014a1619e5d85c1"} Nov 03 23:26:17.061353 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.061318 5051 patch_prober.go:29] interesting pod/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body= Nov 03 23:26:17.061555 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.061372 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" Nov 03 23:26:17.338275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.338247 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:17.459323 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.459291 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\" (UniqueName: \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\") pod \"14caff3e-c495-4c8a-96d3-9b60436a5bda\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " Nov 03 23:26:17.459514 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.459363 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\") pod \"14caff3e-c495-4c8a-96d3-9b60436a5bda\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " Nov 03 23:26:17.459603 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.459536 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\") pod \"14caff3e-c495-4c8a-96d3-9b60436a5bda\" (UID: \"14caff3e-c495-4c8a-96d3-9b60436a5bda\") " Nov 03 23:26:17.459802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.459772 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca" pod "14caff3e-c495-4c8a-96d3-9b60436a5bda" (UID: "14caff3e-c495-4c8a-96d3-9b60436a5bda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:17.460016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.459976 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:17.462177 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.462152 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca" pod "14caff3e-c495-4c8a-96d3-9b60436a5bda" (UID: "14caff3e-c495-4c8a-96d3-9b60436a5bda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:26:17.462300 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.462182 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj" pod "14caff3e-c495-4c8a-96d3-9b60436a5bda" (UID: "14caff3e-c495-4c8a-96d3-9b60436a5bda"). InnerVolumeSpecName "kube-api-access-jq7qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:17.560906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.560881 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14caff3e-c495-4c8a-96d3-9b60436a5bda-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:17.560906 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:17.560906 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7qj\" (UniqueName: \"kubernetes.io/projected/14caff3e-c495-4c8a-96d3-9b60436a5bda-kube-api-access-jq7qj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:18.065650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:18.065622 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:18.066815 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:18.066794 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:18.210093 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:18.210062 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" Nov 03 23:26:18.210093 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:18.210081 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn" event={"ID":"14caff3e-c495-4c8a-96d3-9b60436a5bda","Type":"ContainerDied","Data":"94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281"} Nov 03 23:26:18.210352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:18.210126 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d7fb00e068b963f54bc36ab4240d4a6be7eca9b20b953812790641d596c281" Nov 03 23:26:21.622948 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:26:21.622911 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:26:25.092462 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:25.092424 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:26:25.196750 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:25.196711 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:26:30.093457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.093403 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" podUID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" containerName="payload-processing" containerID="cri-o://3dfe9a63bc563cedc7bbab3b15142888063f7d3bb8025d8970d2ff964a73a0c4" gracePeriod=25 Nov 03 23:26:30.197025 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.196963 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" podUID="cb633fff-35d9-4607-ab4e-db59cb21e860" containerName="payload-pre-processing" containerID="cri-o://8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a" gracePeriod=25 Nov 03 23:26:30.280215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.280180 5051 generic.go:356] "Generic (PLEG): container finished" podID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" containerID="3dfe9a63bc563cedc7bbab3b15142888063f7d3bb8025d8970d2ff964a73a0c4" exitCode=0 Nov 03 23:26:30.280404 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.280242 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" event={"ID":"7ab4a9f1-12fb-4ba4-9260-49ababc545ff","Type":"ContainerDied","Data":"3dfe9a63bc563cedc7bbab3b15142888063f7d3bb8025d8970d2ff964a73a0c4"} Nov 03 23:26:30.356090 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.356057 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:26:30.469786 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.469758 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:26:30.490347 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.490315 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\" (UniqueName: \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\") pod \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " Nov 03 23:26:30.490571 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.490425 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\") pod \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\" (UID: \"7ab4a9f1-12fb-4ba4-9260-49ababc545ff\") " Nov 03 23:26:30.490874 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.490848 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume" pod "7ab4a9f1-12fb-4ba4-9260-49ababc545ff" (UID: "7ab4a9f1-12fb-4ba4-9260-49ababc545ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:30.492810 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.492785 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2" pod "7ab4a9f1-12fb-4ba4-9260-49ababc545ff" (UID: "7ab4a9f1-12fb-4ba4-9260-49ababc545ff"). InnerVolumeSpecName "kube-api-access-5wwk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:30.591209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591170 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\" (UniqueName: \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\") pod \"cb633fff-35d9-4607-ab4e-db59cb21e860\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " Nov 03 23:26:30.591598 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591225 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\") pod \"cb633fff-35d9-4607-ab4e-db59cb21e860\" (UID: \"cb633fff-35d9-4607-ab4e-db59cb21e860\") " Nov 03 23:26:30.591598 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591588 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume" pod "cb633fff-35d9-4607-ab4e-db59cb21e860" (UID: "cb633fff-35d9-4607-ab4e-db59cb21e860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:30.591781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591720 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:30.591781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591739 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb633fff-35d9-4607-ab4e-db59cb21e860-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:30.591781 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.591753 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwk2\" (UniqueName: \"kubernetes.io/projected/7ab4a9f1-12fb-4ba4-9260-49ababc545ff-kube-api-access-5wwk2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:30.594131 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.594101 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl" pod "cb633fff-35d9-4607-ab4e-db59cb21e860" (UID: "cb633fff-35d9-4607-ab4e-db59cb21e860"). InnerVolumeSpecName "kube-api-access-fsvnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:30.693269 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:30.693180 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvnl\" (UniqueName: \"kubernetes.io/projected/cb633fff-35d9-4607-ab4e-db59cb21e860-kube-api-access-fsvnl\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:31.287508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.287466 5051 generic.go:356] "Generic (PLEG): container finished" podID="cb633fff-35d9-4607-ab4e-db59cb21e860" containerID="8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a" exitCode=0 Nov 03 23:26:31.288056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.287539 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" Nov 03 23:26:31.288056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.287558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" event={"ID":"cb633fff-35d9-4607-ab4e-db59cb21e860","Type":"ContainerDied","Data":"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a"} Nov 03 23:26:31.288056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.287670 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj" event={"ID":"cb633fff-35d9-4607-ab4e-db59cb21e860","Type":"ContainerDied","Data":"8d0440e7f30cf5f7cb6fba89cababd42e0e7d8855374409e5748125c58e05c64"} Nov 03 23:26:31.288056 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.287706 5051 scope.go:122] "RemoveContainer" containerID="8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a" Nov 03 23:26:31.289299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.289283 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" Nov 03 23:26:31.289299 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.289285 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk" event={"ID":"7ab4a9f1-12fb-4ba4-9260-49ababc545ff","Type":"ContainerDied","Data":"e1feb922b2469d408f37cea0a7ff711185daf4698bd48ec4545e00efe0516406"} Nov 03 23:26:31.299773 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.299668 5051 scope.go:122] "RemoveContainer" containerID="8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a" Nov 03 23:26:31.300133 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:26:31.300104 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a\": container with ID starting with 8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a not found: ID does not exist" containerID="8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a" Nov 03 23:26:31.300240 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.300143 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a"} err="failed to get container status \"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a\": rpc error: code = NotFound desc = could not find container \"8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a\": container with ID starting with 8c1c11a18ab6fa982f8856c3e07d490f21137ad4439cbd8f573fe557bef4698a not found: ID does not exist" Nov 03 23:26:31.300240 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.300161 5051 scope.go:122] "RemoveContainer" containerID="3dfe9a63bc563cedc7bbab3b15142888063f7d3bb8025d8970d2ff964a73a0c4" Nov 03 23:26:31.452283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.452196 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:26:31.503551 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.503518 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-738d0c9a-76ddd9d799-pnhcj"] Nov 03 23:26:31.568478 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.568437 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:26:31.573492 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:31.573448 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-738d0c9a-57b49dd7f6-9vwgk"] Nov 03 23:26:32.407644 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:32.407612 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" path="/var/lib/kubelet/pods/7ab4a9f1-12fb-4ba4-9260-49ababc545ff/volumes" Nov 03 23:26:32.408048 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:32.408019 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb633fff-35d9-4607-ab4e-db59cb21e860" path="/var/lib/kubelet/pods/cb633fff-35d9-4607-ab4e-db59cb21e860/volumes" Nov 03 23:26:33.659578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:33.659542 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn"] Nov 03 23:26:33.709098 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:33.709061 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-738d0c9a-925cn"] Nov 03 23:26:34.409976 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:34.409934 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14caff3e-c495-4c8a-96d3-9b60436a5bda" path="/var/lib/kubelet/pods/14caff3e-c495-4c8a-96d3-9b60436a5bda/volumes" Nov 03 23:26:34.449075 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:34.449038 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:26:34.449314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:34.449288 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" containerID="cri-o://eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87" gracePeriod=30 Nov 03 23:26:36.192913 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.192865 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:26:36.193582 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193559 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cb633fff-35d9-4607-ab4e-db59cb21e860" containerName="payload-pre-processing" Nov 03 23:26:36.193682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193584 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb633fff-35d9-4607-ab4e-db59cb21e860" containerName="payload-pre-processing" Nov 03 23:26:36.193682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193602 5051 cpu_manager.go:414] "containerMap: removing container" podUID="14caff3e-c495-4c8a-96d3-9b60436a5bda" containerName="revoke-keys" Nov 03 23:26:36.193682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193609 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14caff3e-c495-4c8a-96d3-9b60436a5bda" containerName="revoke-keys" Nov 03 23:26:36.193682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193646 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" containerName="payload-processing" Nov 03 23:26:36.193682 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193653 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" containerName="payload-processing" Nov 03 23:26:36.193955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193768 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb633fff-35d9-4607-ab4e-db59cb21e860" containerName="payload-pre-processing" Nov 03 23:26:36.193955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193783 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="14caff3e-c495-4c8a-96d3-9b60436a5bda" containerName="revoke-keys" Nov 03 23:26:36.193955 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.193797 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab4a9f1-12fb-4ba4-9260-49ababc545ff" containerName="payload-processing" Nov 03 23:26:36.198576 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.198555 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.251324 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.251259 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-034b52-openshift-default-dockercfg-khbct\"" Nov 03 23:26:36.303441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.303405 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:26:36.359245 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359211 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxjp\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359249 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359358 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359380 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359415 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359530 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359572 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359567 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359642 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359632 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.359677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.359662 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.460712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.460614 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.461564 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.461535 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.461768 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.461421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.461898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.461777 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462077 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462056 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462247 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462399 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462502 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462484 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxjp\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462606 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.462838 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462778 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.463121 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.463100 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.463283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.463262 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.463363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.462635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.465394 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.465365 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.471382 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.471351 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.475352 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.475325 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.522856 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.522829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.525097 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.525070 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxjp\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\") pod \"e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:36.811575 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:36.811492 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:37.249248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.249220 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:26:37.252276 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:26:37.252241 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960a591d_7664_4033_8a04_c28e39c7b862.slice/crio-66d9303578ee689880aedab80eef2e3b0afda86594882d2887d9ddb0d0572862 WatchSource:0}: Error finding container 66d9303578ee689880aedab80eef2e3b0afda86594882d2887d9ddb0d0572862: Status 404 returned error can't find the container with id 66d9303578ee689880aedab80eef2e3b0afda86594882d2887d9ddb0d0572862 Nov 03 23:26:37.254943 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.254902 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:37.255057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.255019 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:37.255108 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.255063 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:26:37.327095 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.327060 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerStarted","Data":"fd4db9e1c9b232fb8d9db35ebedbd837ee18cbffa11e30311b587c6ad7065a90"} Nov 03 23:26:37.327219 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.327124 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerStarted","Data":"66d9303578ee689880aedab80eef2e3b0afda86594882d2887d9ddb0d0572862"} Nov 03 23:26:37.447228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.446877 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podStartSLOduration=1.446860312 podStartE2EDuration="1.446860312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:37.446360682 +0000 UTC m=+3871.183704909" watchObservedRunningTime="2026-11-03 23:26:37.446860312 +0000 UTC m=+3871.184204540" Nov 03 23:26:37.812489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:37.812458 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:38.817607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:38.817579 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:39.339623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.339592 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:39.340452 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.340433 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:26:39.708957 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.708934 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:39.797250 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797218 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797280 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797313 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797376 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797425 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797482 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797479 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797525 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797598 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797629 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\") pod \"66e13648-c1f9-41d4-a455-68597db4c4de\" (UID: \"66e13648-c1f9-41d4-a455-68597db4c4de\") " Nov 03 23:26:39.797765 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797646 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:26:39.797981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797799 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:39.797981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797904 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:39.797981 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.797937 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:39.798171 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.798055 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:39.798171 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.798123 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e13648-c1f9-41d4-a455-68597db4c4de-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.798171 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.798142 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.798171 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.798159 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.798340 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.798172 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.800049 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.799965 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:26:39.800158 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.800072 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:39.800510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.800474 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:26:39.800510 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.800504 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj" pod "66e13648-c1f9-41d4-a455-68597db4c4de" (UID: "66e13648-c1f9-41d4-a455-68597db4c4de"). InnerVolumeSpecName "kube-api-access-rvvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:26:39.899033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.898923 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvqj\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-kube-api-access-rvvqj\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.899033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.898952 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e13648-c1f9-41d4-a455-68597db4c4de-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.899033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.898961 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.899033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.898970 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e13648-c1f9-41d4-a455-68597db4c4de-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:39.899033 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:39.898978 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e13648-c1f9-41d4-a455-68597db4c4de-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:26:40.346623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.346540 5051 generic.go:356] "Generic (PLEG): container finished" podID="66e13648-c1f9-41d4-a455-68597db4c4de" containerID="eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87" exitCode=0 Nov 03 23:26:40.346623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.346590 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" Nov 03 23:26:40.346842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.346624 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerDied","Data":"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87"} Nov 03 23:26:40.346842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.346685 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs" event={"ID":"66e13648-c1f9-41d4-a455-68597db4c4de","Type":"ContainerDied","Data":"82070c232f4f341d266124460a39ab625d5ce311adaac6711f1510c1525b7141"} Nov 03 23:26:40.346842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.346707 5051 scope.go:122] "RemoveContainer" containerID="eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87" Nov 03 23:26:40.362947 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.362922 5051 scope.go:122] "RemoveContainer" containerID="879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410" Nov 03 23:26:40.372689 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.372644 5051 scope.go:122] "RemoveContainer" containerID="eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87" Nov 03 23:26:40.373114 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:26:40.373082 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87\": container with ID starting with eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87 not found: ID does not exist" containerID="eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87" Nov 03 23:26:40.373209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.373134 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87"} err="failed to get container status \"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87\": rpc error: code = NotFound desc = could not find container \"eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87\": container with ID starting with eb2db7433e8a6219ac0a219345657bdbeaca4f5be254587ec458399088df6c87 not found: ID does not exist" Nov 03 23:26:40.373209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.373169 5051 scope.go:122] "RemoveContainer" containerID="879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410" Nov 03 23:26:40.373586 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:26:40.373543 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410\": container with ID starting with 879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410 not found: ID does not exist" containerID="879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410" Nov 03 23:26:40.373657 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.373597 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410"} err="failed to get container status \"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410\": rpc error: code = NotFound desc = could not find container \"879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410\": container with ID starting with 879052ad10a483fb71fa0953577dddb0cd72f486dd70caa129c73681bc550410 not found: ID does not exist" Nov 03 23:26:40.512404 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.512372 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:26:40.562517 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:40.562484 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-738d0c9a-openshift-default-6dd5576d67-lcmqs"] Nov 03 23:26:42.407368 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:42.407335 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" path="/var/lib/kubelet/pods/66e13648-c1f9-41d4-a455-68597db4c4de/volumes" Nov 03 23:26:43.884067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.884041 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:26:43.884505 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.884490 5051 cpu_manager.go:414] "containerMap: removing container" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:43.884557 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.884507 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:43.884629 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.884616 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:43.889148 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.889131 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:43.941525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.941497 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-034b52\"" Nov 03 23:26:43.943274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.943244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmj8\" (UniqueName: \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:43.943429 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.943323 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:43.986091 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.986059 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:26:43.991057 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:43.991032 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-034b52-dockercfg-vbxmv\"" Nov 03 23:26:44.043213 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.043179 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:26:44.043690 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.043672 5051 cpu_manager.go:414] "containerMap: removing container" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:44.043802 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.043694 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:44.043863 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.043829 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e13648-c1f9-41d4-a455-68597db4c4de" containerName="istio-proxy" Nov 03 23:26:44.044463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.044435 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmj8\" (UniqueName: \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:44.044596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.044486 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:44.045167 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.045146 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:44.047216 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.047199 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.103471 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.103443 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:26:44.109031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.107854 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmj8\" (UniqueName: \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\") pod \"payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:44.145081 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.144980 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.145247 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.145121 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7lv\" (UniqueName: \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.202540 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.202506 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:44.246109 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.246075 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.246273 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.246144 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7lv\" (UniqueName: \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.246778 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.246758 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.307264 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.307226 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7lv\" (UniqueName: \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\") pod \"payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.361591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.361563 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:44.587181 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.587157 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:26:44.589107 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:26:44.589080 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a73878_fd21_42d4_956c_3027d550b57e.slice/crio-e64aefdeec65ee92ed334d26ce34aa468c0eb50a1172fab88094d18ca504125c WatchSource:0}: Error finding container e64aefdeec65ee92ed334d26ce34aa468c0eb50a1172fab88094d18ca504125c: Status 404 returned error can't find the container with id e64aefdeec65ee92ed334d26ce34aa468c0eb50a1172fab88094d18ca504125c Nov 03 23:26:44.744427 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:44.744399 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:26:44.746186 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:26:44.746157 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4045e4ba_1d27_4e08_877a_b5e64ba7c737.slice/crio-921aceb9c5505e35af65d59f020cd458a3df8e4b42c7db7aa94e06fa0aed1814 WatchSource:0}: Error finding container 921aceb9c5505e35af65d59f020cd458a3df8e4b42c7db7aa94e06fa0aed1814: Status 404 returned error can't find the container with id 921aceb9c5505e35af65d59f020cd458a3df8e4b42c7db7aa94e06fa0aed1814 Nov 03 23:26:45.383481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.383444 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" event={"ID":"d4a73878-fd21-42d4-956c-3027d550b57e","Type":"ContainerStarted","Data":"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4"} Nov 03 23:26:45.383928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.383524 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" event={"ID":"d4a73878-fd21-42d4-956c-3027d550b57e","Type":"ContainerStarted","Data":"e64aefdeec65ee92ed334d26ce34aa468c0eb50a1172fab88094d18ca504125c"} Nov 03 23:26:45.383928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.383567 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:45.385301 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.385279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" event={"ID":"4045e4ba-1d27-4e08-877a-b5e64ba7c737","Type":"ContainerStarted","Data":"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233"} Nov 03 23:26:45.385431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.385318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" event={"ID":"4045e4ba-1d27-4e08-877a-b5e64ba7c737","Type":"ContainerStarted","Data":"921aceb9c5505e35af65d59f020cd458a3df8e4b42c7db7aa94e06fa0aed1814"} Nov 03 23:26:45.385431 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.385340 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:26:45.499437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.499378 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" podStartSLOduration=2.499333606 podStartE2EDuration="2.499333606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:45.497920653 +0000 UTC m=+3879.235264879" watchObservedRunningTime="2026-11-03 23:26:45.499333606 +0000 UTC m=+3879.236677831" Nov 03 23:26:45.618370 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:45.618302 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" podStartSLOduration=2.618280941 podStartE2EDuration="2.618280941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:45.613140129 +0000 UTC m=+3879.350484355" watchObservedRunningTime="2026-11-03 23:26:45.618280941 +0000 UTC m=+3879.355625167" Nov 03 23:26:51.626734 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:26:51.626693 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:26:56.394486 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:56.394446 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:26:56.394963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:26:56.394528 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:27:00.478525 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.478489 5051 generic.go:356] "Generic (PLEG): container finished" podID="960a591d-7664-4033-8a04-c28e39c7b862" containerID="fd4db9e1c9b232fb8d9db35ebedbd837ee18cbffa11e30311b587c6ad7065a90" exitCode=0 Nov 03 23:27:00.479087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.478568 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerDied","Data":"fd4db9e1c9b232fb8d9db35ebedbd837ee18cbffa11e30311b587c6ad7065a90"} Nov 03 23:27:00.479087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.478965 5051 scope.go:122] "RemoveContainer" containerID="fd4db9e1c9b232fb8d9db35ebedbd837ee18cbffa11e30311b587c6ad7065a90" Nov 03 23:27:00.480216 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.480178 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:27:00.480314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.480267 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:27:00.480314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:00.480302 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:27:01.489146 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:01.489114 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerStarted","Data":"71f03a6189205a1848acb3dafdb7d79dcfe0a857549112ca41618b0d6b2dab30"} Nov 03 23:27:01.812678 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:01.812586 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:27:01.812898 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:01.812874 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 23:27:01.812945 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:01.812924 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 23:27:02.812034 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:02.811975 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 23:27:02.812497 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:02.812048 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 23:27:03.812887 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:03.812845 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 23:27:03.813389 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:03.812925 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 23:27:04.812530 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:04.812489 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 23:27:04.812727 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:04.812539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 23:27:05.812855 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:05.812821 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 23:27:05.813303 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:05.812885 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 23:27:06.815677 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:06.815650 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:27:06.816101 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:06.816030 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:27:06.816628 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:06.816612 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:27:07.436979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:07.436954 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:27:07.468814 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:07.468786 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:27:12.596178 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:12.596145 5051 scope.go:122] "RemoveContainer" containerID="9e1a5b335ae9a5ed8163d89c51c5c1ed0ebd213f12d85a2ea29a844a7c1fb42c" Nov 03 23:27:12.605903 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:12.605882 5051 scope.go:122] "RemoveContainer" containerID="1e3b1c226903c1e92aeae02f6ad012e5965c96fef65a46539a4f4a6327f7f6ae" Nov 03 23:27:21.623101 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:27:21.623016 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:27:26.050822 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:26.050789 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs"] Nov 03 23:27:26.100228 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:26.100194 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4892429a-zk9fs"] Nov 03 23:27:26.406621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:27:26.406588 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbed8942-51ce-4a61-a5a5-7f73db286281" path="/var/lib/kubelet/pods/dbed8942-51ce-4a61-a5a5-7f73db286281/volumes" Nov 03 23:27:51.623191 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:27:51.623153 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:28:21.623866 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:28:21.623829 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:28:35.702407 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.702370 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r"] Nov 03 23:28:35.706286 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.706269 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.762963 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.762930 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r"] Nov 03 23:28:35.809274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.809247 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mjvcl\"" Nov 03 23:28:35.809274 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.809269 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:28:35.847597 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.847560 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.847749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.847736 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.847797 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.847769 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9pg\" (UniqueName: \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.949480 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.949439 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.949637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.949487 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9pg\" (UniqueName: \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.949637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.949542 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.950214 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.950196 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:35.951948 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:35.951921 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:36.010283 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:36.010219 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9pg\" (UniqueName: \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\") pod \"maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:36.018604 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:36.018578 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:36.408496 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:28:36.408469 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493a6a3e_f67a_4abb_bda6_5500efae14c5.slice/crio-dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74 WatchSource:0}: Error finding container dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74: Status 404 returned error can't find the container with id dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74 Nov 03 23:28:36.410091 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:36.410067 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r"] Nov 03 23:28:37.038821 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:37.038733 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" event={"ID":"493a6a3e-f67a-4abb-bda6-5500efae14c5","Type":"ContainerStarted","Data":"2978c2f7012f9ef653ef550d2ab4a22be296a64f2092ad630e27d9b8df8f52c2"} Nov 03 23:28:37.038821 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:37.038795 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" event={"ID":"493a6a3e-f67a-4abb-bda6-5500efae14c5","Type":"ContainerStarted","Data":"dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74"} Nov 03 23:28:37.154837 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:37.154789 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" podStartSLOduration=2.154776486 podStartE2EDuration="2.154776486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:37.152608183 +0000 UTC m=+3990.889952408" watchObservedRunningTime="2026-11-03 23:28:37.154776486 +0000 UTC m=+3990.892120737" Nov 03 23:28:38.046860 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:38.046819 5051 generic.go:356] "Generic (PLEG): container finished" podID="493a6a3e-f67a-4abb-bda6-5500efae14c5" containerID="2978c2f7012f9ef653ef550d2ab4a22be296a64f2092ad630e27d9b8df8f52c2" exitCode=0 Nov 03 23:28:38.047325 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:38.046906 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" event={"ID":"493a6a3e-f67a-4abb-bda6-5500efae14c5","Type":"ContainerDied","Data":"2978c2f7012f9ef653ef550d2ab4a22be296a64f2092ad630e27d9b8df8f52c2"} Nov 03 23:28:39.197935 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.197910 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:39.283470 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.283442 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\") pod \"493a6a3e-f67a-4abb-bda6-5500efae14c5\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " Nov 03 23:28:39.283626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.283485 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\" (UniqueName: \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\") pod \"493a6a3e-f67a-4abb-bda6-5500efae14c5\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " Nov 03 23:28:39.283626 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.283515 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\") pod \"493a6a3e-f67a-4abb-bda6-5500efae14c5\" (UID: \"493a6a3e-f67a-4abb-bda6-5500efae14c5\") " Nov 03 23:28:39.283790 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.283771 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca" pod "493a6a3e-f67a-4abb-bda6-5500efae14c5" (UID: "493a6a3e-f67a-4abb-bda6-5500efae14c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:28:39.283905 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.283880 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:39.285713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.285692 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca" pod "493a6a3e-f67a-4abb-bda6-5500efae14c5" (UID: "493a6a3e-f67a-4abb-bda6-5500efae14c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:28:39.285856 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.285837 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg" pod "493a6a3e-f67a-4abb-bda6-5500efae14c5" (UID: "493a6a3e-f67a-4abb-bda6-5500efae14c5"). InnerVolumeSpecName "kube-api-access-gh9pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:28:39.384888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.384853 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9pg\" (UniqueName: \"kubernetes.io/projected/493a6a3e-f67a-4abb-bda6-5500efae14c5-kube-api-access-gh9pg\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:39.384888 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:39.384879 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493a6a3e-f67a-4abb-bda6-5500efae14c5-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:40.060205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:40.060171 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" Nov 03 23:28:40.060205 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:40.060196 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r" event={"ID":"493a6a3e-f67a-4abb-bda6-5500efae14c5","Type":"ContainerDied","Data":"dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74"} Nov 03 23:28:40.060428 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:40.060239 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc32e6aa50df4cf48b34742d75371af6caf798979af4dc32be2f62216366bf74" Nov 03 23:28:47.224239 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:47.224160 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:28:47.463136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:47.463100 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:28:51.623347 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:28:51.623310 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:28:52.225273 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.225231 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" podUID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" containerName="payload-processing" containerID="cri-o://d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233" gracePeriod=25 Nov 03 23:28:52.463674 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.463618 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" podUID="d4a73878-fd21-42d4-956c-3027d550b57e" containerName="payload-pre-processing" containerID="cri-o://62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4" gracePeriod=25 Nov 03 23:28:52.541670 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.541646 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:28:52.622267 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.622236 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\") pod \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " Nov 03 23:28:52.622434 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.622329 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\" (UniqueName: \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\") pod \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\" (UID: \"4045e4ba-1d27-4e08-877a-b5e64ba7c737\") " Nov 03 23:28:52.622680 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.622654 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume" pod "4045e4ba-1d27-4e08-877a-b5e64ba7c737" (UID: "4045e4ba-1d27-4e08-877a-b5e64ba7c737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:28:52.624621 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.624586 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv" pod "4045e4ba-1d27-4e08-877a-b5e64ba7c737" (UID: "4045e4ba-1d27-4e08-877a-b5e64ba7c737"). InnerVolumeSpecName "kube-api-access-nz7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:28:52.709602 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.709578 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:28:52.723719 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.723693 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\" (UniqueName: \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\") pod \"d4a73878-fd21-42d4-956c-3027d550b57e\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " Nov 03 23:28:52.723846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.723820 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\") pod \"d4a73878-fd21-42d4-956c-3027d550b57e\" (UID: \"d4a73878-fd21-42d4-956c-3027d550b57e\") " Nov 03 23:28:52.724190 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.724164 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume" pod "d4a73878-fd21-42d4-956c-3027d550b57e" (UID: "d4a73878-fd21-42d4-956c-3027d550b57e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:28:52.724318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.724203 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4045e4ba-1d27-4e08-877a-b5e64ba7c737-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:52.724318 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.724225 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7lv\" (UniqueName: \"kubernetes.io/projected/4045e4ba-1d27-4e08-877a-b5e64ba7c737-kube-api-access-nz7lv\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:52.725979 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.725962 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8" pod "d4a73878-fd21-42d4-956c-3027d550b57e" (UID: "d4a73878-fd21-42d4-956c-3027d550b57e"). InnerVolumeSpecName "kube-api-access-lzmj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:28:52.825606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.825510 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzmj8\" (UniqueName: \"kubernetes.io/projected/d4a73878-fd21-42d4-956c-3027d550b57e-kube-api-access-lzmj8\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:52.825606 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:52.825550 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a73878-fd21-42d4-956c-3027d550b57e-plugins-config-volume\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:28:53.132777 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.132743 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4a73878-fd21-42d4-956c-3027d550b57e" containerID="62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4" exitCode=0 Nov 03 23:28:53.132985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.132792 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" Nov 03 23:28:53.132985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.132834 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" event={"ID":"d4a73878-fd21-42d4-956c-3027d550b57e","Type":"ContainerDied","Data":"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4"} Nov 03 23:28:53.132985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.132893 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x" event={"ID":"d4a73878-fd21-42d4-956c-3027d550b57e","Type":"ContainerDied","Data":"e64aefdeec65ee92ed334d26ce34aa468c0eb50a1172fab88094d18ca504125c"} Nov 03 23:28:53.132985 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.132929 5051 scope.go:122] "RemoveContainer" containerID="62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4" Nov 03 23:28:53.134349 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.134323 5051 generic.go:356] "Generic (PLEG): container finished" podID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" containerID="d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233" exitCode=0 Nov 03 23:28:53.134460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.134367 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" event={"ID":"4045e4ba-1d27-4e08-877a-b5e64ba7c737","Type":"ContainerDied","Data":"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233"} Nov 03 23:28:53.134460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.134408 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" event={"ID":"4045e4ba-1d27-4e08-877a-b5e64ba7c737","Type":"ContainerDied","Data":"921aceb9c5505e35af65d59f020cd458a3df8e4b42c7db7aa94e06fa0aed1814"} Nov 03 23:28:53.134460 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.134373 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52" Nov 03 23:28:53.144151 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.144134 5051 scope.go:122] "RemoveContainer" containerID="62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4" Nov 03 23:28:53.144464 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:28:53.144431 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4\": container with ID starting with 62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4 not found: ID does not exist" containerID="62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4" Nov 03 23:28:53.144535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.144465 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4"} err="failed to get container status \"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4\": rpc error: code = NotFound desc = could not find container \"62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4\": container with ID starting with 62230ef5185fbbb205a56b0ad090bb055302d50e98d343a125ee01fac63df4b4 not found: ID does not exist" Nov 03 23:28:53.144535 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.144482 5051 scope.go:122] "RemoveContainer" containerID="d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233" Nov 03 23:28:53.152829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.152813 5051 scope.go:122] "RemoveContainer" containerID="d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233" Nov 03 23:28:53.153101 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:28:53.153083 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233\": container with ID starting with d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233 not found: ID does not exist" containerID="d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233" Nov 03 23:28:53.153179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.153108 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233"} err="failed to get container status \"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233\": rpc error: code = NotFound desc = could not find container \"d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233\": container with ID starting with d3e4c3c25ae41be103ac2f6ef5af26374d7f0d9a1e4e1d25401d7b282e0aa233 not found: ID does not exist" Nov 03 23:28:53.296904 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.296875 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:28:53.346328 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.346301 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-034b52-555f758647-zv96x"] Nov 03 23:28:53.454565 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.454487 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:28:53.458417 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:53.458390 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-034b52-7c869c85b4-57j52"] Nov 03 23:28:54.407780 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:54.407752 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" path="/var/lib/kubelet/pods/4045e4ba-1d27-4e08-877a-b5e64ba7c737/volumes" Nov 03 23:28:54.408184 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:54.408148 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a73878-fd21-42d4-956c-3027d550b57e" path="/var/lib/kubelet/pods/d4a73878-fd21-42d4-956c-3027d550b57e/volumes" Nov 03 23:28:56.176346 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:56.176316 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r"] Nov 03 23:28:56.225725 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:56.225696 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-034b52-dmc7r"] Nov 03 23:28:56.408596 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:28:56.408565 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493a6a3e-f67a-4abb-bda6-5500efae14c5" path="/var/lib/kubelet/pods/493a6a3e-f67a-4abb-bda6-5500efae14c5/volumes" Nov 03 23:29:03.082736 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:03.082697 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:29:03.083206 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:03.082927 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" containerID="cri-o://71f03a6189205a1848acb3dafdb7d79dcfe0a857549112ca41618b0d6b2dab30" gracePeriod=30 Nov 03 23:29:06.818507 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:06.818474 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:29:06.818983 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:06.818530 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:29:08.226137 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.226104 5051 generic.go:356] "Generic (PLEG): container finished" podID="960a591d-7664-4033-8a04-c28e39c7b862" containerID="71f03a6189205a1848acb3dafdb7d79dcfe0a857549112ca41618b0d6b2dab30" exitCode=0 Nov 03 23:29:08.226622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.226180 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerDied","Data":"71f03a6189205a1848acb3dafdb7d79dcfe0a857549112ca41618b0d6b2dab30"} Nov 03 23:29:08.226622 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.226260 5051 scope.go:122] "RemoveContainer" containerID="fd4db9e1c9b232fb8d9db35ebedbd837ee18cbffa11e30311b587c6ad7065a90" Nov 03 23:29:08.346139 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.346076 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:29:08.479594 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479564 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.479594 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479601 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.479923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479623 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.479923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479820 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.479923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479864 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.479923 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479899 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.480198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479928 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:29:08.480198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479939 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:29:08.480198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.479932 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.480198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480104 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.480198 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480185 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:08.480477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480190 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\") pod \"960a591d-7664-4033-8a04-c28e39c7b862\" (UID: \"960a591d-7664-4033-8a04-c28e39c7b862\") " Nov 03 23:29:08.480477 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480217 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:29:08.480585 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480546 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:29:08.481069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480952 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.481069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.480979 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-data\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.481069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.481012 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-workload-certs\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.481069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.481027 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960a591d-7664-4033-8a04-c28e39c7b862-istiod-ca-cert\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.481069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.481042 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-credential-socket\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.482506 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.482448 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:29:08.482639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.482595 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:08.482700 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.482660 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "kube-api-access-kzxjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:08.482842 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.482824 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo" pod "960a591d-7664-4033-8a04-c28e39c7b862" (UID: "960a591d-7664-4033-8a04-c28e39c7b862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:29:08.582463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.582430 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-istio-token\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.582463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.582461 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxjp\" (UniqueName: \"kubernetes.io/projected/960a591d-7664-4033-8a04-c28e39c7b862-kube-api-access-kzxjp\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.582463 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.582474 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960a591d-7664-4033-8a04-c28e39c7b862-istio-podinfo\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:08.582723 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:08.582483 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960a591d-7664-4033-8a04-c28e39c7b862-istio-envoy\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:09.233179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:09.233145 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" event={"ID":"960a591d-7664-4033-8a04-c28e39c7b862","Type":"ContainerDied","Data":"66d9303578ee689880aedab80eef2e3b0afda86594882d2887d9ddb0d0572862"} Nov 03 23:29:09.233179 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:09.233162 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx" Nov 03 23:29:09.233721 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:09.233206 5051 scope.go:122] "RemoveContainer" containerID="71f03a6189205a1848acb3dafdb7d79dcfe0a857549112ca41618b0d6b2dab30" Nov 03 23:29:09.450321 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:09.450288 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:29:09.499093 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:09.498982 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-034b52-openshift-default-546fbd7649-76chx"] Nov 03 23:29:10.407467 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:10.407423 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960a591d-7664-4033-8a04-c28e39c7b862" path="/var/lib/kubelet/pods/960a591d-7664-4033-8a04-c28e39c7b862/volumes" Nov 03 23:29:12.652501 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:12.652458 5051 scope.go:122] "RemoveContainer" containerID="023a75f9184008b4dc163ddaa7547c2a15db6d86943f9653ac254a0c081899cd" Nov 03 23:29:12.661563 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:12.661542 5051 scope.go:122] "RemoveContainer" containerID="6249fd56ce0860bcf2a67b02afafa17e592fcec0e93b3a889b08a813b7f842f4" Nov 03 23:29:21.622256 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:29:21.622217 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:29:28.046853 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:28.046801 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj"] Nov 03 23:29:28.095833 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:28.095801 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qvsfj"] Nov 03 23:29:28.406656 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:28.406625 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802f9f2b-9d3f-41e9-9886-fd2406d28465" path="/var/lib/kubelet/pods/802f9f2b-9d3f-41e9-9886-fd2406d28465/volumes" Nov 03 23:29:30.422387 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422351 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8"] Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422745 5051 cpu_manager.go:414] "containerMap: removing container" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422755 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422769 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" containerName="payload-processing" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422775 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" containerName="payload-processing" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422786 5051 cpu_manager.go:414] "containerMap: removing container" podUID="493a6a3e-f67a-4abb-bda6-5500efae14c5" containerName="revoke-keys" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422792 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a6a3e-f67a-4abb-bda6-5500efae14c5" containerName="revoke-keys" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422813 5051 cpu_manager.go:414] "containerMap: removing container" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.422818 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422818 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422832 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4a73878-fd21-42d4-956c-3027d550b57e" containerName="payload-pre-processing" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422837 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a73878-fd21-42d4-956c-3027d550b57e" containerName="payload-pre-processing" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422899 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422907 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a73878-fd21-42d4-956c-3027d550b57e" containerName="payload-pre-processing" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422916 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="960a591d-7664-4033-8a04-c28e39c7b862" containerName="istio-proxy" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422922 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4045e4ba-1d27-4e08-877a-b5e64ba7c737" containerName="payload-processing" Nov 03 23:29:30.423150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.422928 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a6a3e-f67a-4abb-bda6-5500efae14c5" containerName="revoke-keys" Nov 03 23:29:30.427511 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.427492 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.472211 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.472179 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.472424 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.472247 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstnx\" (UniqueName: \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.472424 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.472347 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.484601 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.484570 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8"] Nov 03 23:29:30.530420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.530389 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:29:30.530420 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.530396 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mjvcl\"" Nov 03 23:29:30.572865 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.572821 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.573118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.572918 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstnx\" (UniqueName: \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.573118 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.572956 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.573717 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.573695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.575537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.575511 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.643328 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.643287 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstnx\" (UniqueName: \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\") pod \"maas-api-revoke-sub-f03629db-k7jk8\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:30.741292 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:30.741194 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:31.123398 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:31.123371 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8"] Nov 03 23:29:31.125882 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:29:31.125854 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f01b39f_3ac7_40df_a7fa_d1118c215f3d.slice/crio-98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852 WatchSource:0}: Error finding container 98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852: Status 404 returned error can't find the container with id 98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852 Nov 03 23:29:31.366978 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:31.366945 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" event={"ID":"0f01b39f-3ac7-40df-a7fa-d1118c215f3d","Type":"ContainerStarted","Data":"98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852"} Nov 03 23:29:32.373752 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:32.373717 5051 generic.go:356] "Generic (PLEG): container finished" podID="0f01b39f-3ac7-40df-a7fa-d1118c215f3d" containerID="0d25644603a7e12da7581137daf9f2a164f1d26b50091adda1f71fba94bfd705" exitCode=0 Nov 03 23:29:32.374168 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:32.373797 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" event={"ID":"0f01b39f-3ac7-40df-a7fa-d1118c215f3d","Type":"ContainerDied","Data":"0d25644603a7e12da7581137daf9f2a164f1d26b50091adda1f71fba94bfd705"} Nov 03 23:29:33.529867 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.529842 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:33.601821 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.601791 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\") pod \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " Nov 03 23:29:33.602022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.601881 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\" (UniqueName: \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\") pod \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " Nov 03 23:29:33.602022 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.601912 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\") pod \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\" (UID: \"0f01b39f-3ac7-40df-a7fa-d1118c215f3d\") " Nov 03 23:29:33.602340 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.602316 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca" pod "0f01b39f-3ac7-40df-a7fa-d1118c215f3d" (UID: "0f01b39f-3ac7-40df-a7fa-d1118c215f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:33.604221 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.604204 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca" pod "0f01b39f-3ac7-40df-a7fa-d1118c215f3d" (UID: "0f01b39f-3ac7-40df-a7fa-d1118c215f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:33.604290 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.604234 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx" pod "0f01b39f-3ac7-40df-a7fa-d1118c215f3d" (UID: "0f01b39f-3ac7-40df-a7fa-d1118c215f3d"). InnerVolumeSpecName "kube-api-access-jstnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:33.702847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.702755 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jstnx\" (UniqueName: \"kubernetes.io/projected/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-kube-api-access-jstnx\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:33.702847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.702789 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:33.702847 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:33.702800 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f01b39f-3ac7-40df-a7fa-d1118c215f3d-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:34.386282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:34.386251 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" Nov 03 23:29:34.386282 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:34.386261 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k7jk8" event={"ID":"0f01b39f-3ac7-40df-a7fa-d1118c215f3d","Type":"ContainerDied","Data":"98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852"} Nov 03 23:29:34.386552 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:34.386308 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98fbc8ff08a56a8b91deb527c62c136cd00ec75b3f182af1ad5181cdc71b9852" Nov 03 23:29:49.585114 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.585073 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k"] Nov 03 23:29:49.608713 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.608680 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0f01b39f-3ac7-40df-a7fa-d1118c215f3d" containerName="revoke-keys" Nov 03 23:29:49.608936 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.608918 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f01b39f-3ac7-40df-a7fa-d1118c215f3d" containerName="revoke-keys" Nov 03 23:29:49.609215 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.609196 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f01b39f-3ac7-40df-a7fa-d1118c215f3d" containerName="revoke-keys" Nov 03 23:29:49.638422 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.638382 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.647637 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.646918 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k"] Nov 03 23:29:49.729881 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.729852 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:29:49.779319 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.779289 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mjvcl\"" Nov 03 23:29:49.808024 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.797528 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qf2\" (UniqueName: \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.808314 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.808291 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.808591 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.808568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.909605 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.909562 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.910743 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.910622 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.914120 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.914096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qf2\" (UniqueName: \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.918103 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.918073 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.927067 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.927039 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:49.981016 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:49.980765 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qf2\" (UniqueName: \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\") pod \"maas-api-revoke-sub-0165d7d2-dc72k\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:50.279069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:50.278152 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:50.848560 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:50.848456 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k"] Nov 03 23:29:50.853315 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:29:50.853283 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4e4b9a_7274_468b_8dd6_5ef953cb8cdd.slice/crio-a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc WatchSource:0}: Error finding container a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc: Status 404 returned error can't find the container with id a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc Nov 03 23:29:51.541114 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:51.541026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" event={"ID":"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd","Type":"ContainerStarted","Data":"a14d9f00a11b2e1538adf0c34e211d87b226653073c36f38c9c85e6bf74b13b7"} Nov 03 23:29:51.541114 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:51.541089 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" event={"ID":"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd","Type":"ContainerStarted","Data":"a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc"} Nov 03 23:29:51.622873 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:29:51.622835 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:29:51.656213 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:51.656151 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" podStartSLOduration=2.656131622 podStartE2EDuration="2.656131622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:51.6555941 +0000 UTC m=+4065.392938326" watchObservedRunningTime="2026-11-03 23:29:51.656131622 +0000 UTC m=+4065.393475848" Nov 03 23:29:52.547461 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:52.547384 5051 generic.go:356] "Generic (PLEG): container finished" podID="cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" containerID="a14d9f00a11b2e1538adf0c34e211d87b226653073c36f38c9c85e6bf74b13b7" exitCode=0 Nov 03 23:29:52.547886 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:52.547526 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" event={"ID":"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd","Type":"ContainerDied","Data":"a14d9f00a11b2e1538adf0c34e211d87b226653073c36f38c9c85e6bf74b13b7"} Nov 03 23:29:53.692615 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.692591 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:53.707846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.707819 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\") pod \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " Nov 03 23:29:53.707951 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.707871 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\") pod \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " Nov 03 23:29:53.708053 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.708009 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\" (UniqueName: \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\") pod \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\" (UID: \"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd\") " Nov 03 23:29:53.708203 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.708179 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca" pod "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" (UID: "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:29:53.708334 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.708319 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.710079 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.710050 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca" pod "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" (UID: "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:29:53.710176 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.710122 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2" pod "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" (UID: "cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd"). InnerVolumeSpecName "kube-api-access-45qf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:29:53.809074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.808952 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qf2\" (UniqueName: \"kubernetes.io/projected/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-kube-api-access-45qf2\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:53.809074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:53.809019 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:29:54.561412 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:54.561374 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" event={"ID":"cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd","Type":"ContainerDied","Data":"a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc"} Nov 03 23:29:54.561619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:54.561418 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dc72k" Nov 03 23:29:54.561619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:54.561422 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a634f3cf2fc1c97c0e87d799d2976159068364a50c5689557eabc269d7e278fc" Nov 03 23:29:58.045741 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:58.045703 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89"] Nov 03 23:29:58.048607 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:58.048580 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qxh89"] Nov 03 23:29:58.405712 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:29:58.405671 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e33a56-418c-42af-8333-4223d934bef6" path="/var/lib/kubelet/pods/e9e33a56-418c-42af-8333-4223d934bef6/volumes" Nov 03 23:30:00.165485 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.165441 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:00.165901 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.165888 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" containerName="revoke-keys" Nov 03 23:30:00.165941 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.165902 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" containerName="revoke-keys" Nov 03 23:30:00.166050 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.166040 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd" containerName="revoke-keys" Nov 03 23:30:00.169321 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.169307 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.269448 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.269416 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.269641 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.269485 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8gc\" (UniqueName: \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.273508 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.273479 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mjvcl\"" Nov 03 23:30:00.275902 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.275883 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:00.370928 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.370888 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.371141 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.370980 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8gc\" (UniqueName: \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.371651 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.371630 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.432103 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.432036 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8gc\" (UniqueName: \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\") pod \"maas-api-key-cleanup-29895810-ksjgc\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.483457 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.483431 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:00.864636 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:00.864494 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:00.867213 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:30:00.867190 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02cc4b3_8541_46b2_80ab_2e9b5c6fb2ea.slice/crio-e440984f4d1c4d37c42193f268b960cde7ced215f566a377f028c5e6a494dda9 WatchSource:0}: Error finding container e440984f4d1c4d37c42193f268b960cde7ced215f566a377f028c5e6a494dda9: Status 404 returned error can't find the container with id e440984f4d1c4d37c42193f268b960cde7ced215f566a377f028c5e6a494dda9 Nov 03 23:30:01.264429 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.264395 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt"] Nov 03 23:30:01.267982 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.267965 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.323062 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.323014 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:30:01.326516 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.326486 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt"] Nov 03 23:30:01.381346 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.381312 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.381537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.381430 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.381537 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.381504 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdnc\" (UniqueName: \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.482676 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.482592 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.482676 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.482638 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdnc\" (UniqueName: \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.482896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.482730 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.483397 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.483377 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.485350 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.485321 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.543542 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.543516 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdnc\" (UniqueName: \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\") pod \"maas-api-revoke-sub-6b19338f-n7ndt\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.581142 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.581106 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:01.602623 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.602595 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerStarted","Data":"a3ea37bfb7d8cbab717c4d67e3dcad3122c5b85b7633e42e9d51bbaa22e0d37c"} Nov 03 23:30:01.602749 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.602645 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerStarted","Data":"e440984f4d1c4d37c42193f268b960cde7ced215f566a377f028c5e6a494dda9"} Nov 03 23:30:01.719294 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.719242 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" podStartSLOduration=1.719228236 podStartE2EDuration="1.719228236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:01.716441598 +0000 UTC m=+4075.453785837" watchObservedRunningTime="2026-11-03 23:30:01.719228236 +0000 UTC m=+4075.456572458" Nov 03 23:30:01.958361 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:01.958332 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt"] Nov 03 23:30:01.961613 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:30:01.961578 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6718947d_9032_4f25_ab38_8e006351cd70.slice/crio-5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f WatchSource:0}: Error finding container 5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f: Status 404 returned error can't find the container with id 5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f Nov 03 23:30:02.609639 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.609607 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" event={"ID":"6718947d-9032-4f25-ab38-8e006351cd70","Type":"ContainerStarted","Data":"0b5bba51101271b0d75230492eb42057a4f5db9690211ef36249c40444012a3f"} Nov 03 23:30:02.610248 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.609659 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" event={"ID":"6718947d-9032-4f25-ab38-8e006351cd70","Type":"ContainerStarted","Data":"5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f"} Nov 03 23:30:02.611336 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.611310 5051 generic.go:356] "Generic (PLEG): container finished" podID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerID="a3ea37bfb7d8cbab717c4d67e3dcad3122c5b85b7633e42e9d51bbaa22e0d37c" exitCode=60 Nov 03 23:30:02.611471 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.611386 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerDied","Data":"a3ea37bfb7d8cbab717c4d67e3dcad3122c5b85b7633e42e9d51bbaa22e0d37c"} Nov 03 23:30:02.611578 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.611563 5051 scope.go:122] "RemoveContainer" containerID="a3ea37bfb7d8cbab717c4d67e3dcad3122c5b85b7633e42e9d51bbaa22e0d37c" Nov 03 23:30:02.726242 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:02.726186 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" podStartSLOduration=1.7261710799999999 podStartE2EDuration="1.72617108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:02.724307213 +0000 UTC m=+4076.461651439" watchObservedRunningTime="2026-11-03 23:30:02.72617108 +0000 UTC m=+4076.463515315" Nov 03 23:30:03.618894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.618808 5051 generic.go:356] "Generic (PLEG): container finished" podID="6718947d-9032-4f25-ab38-8e006351cd70" containerID="0b5bba51101271b0d75230492eb42057a4f5db9690211ef36249c40444012a3f" exitCode=0 Nov 03 23:30:03.619386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.618902 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" event={"ID":"6718947d-9032-4f25-ab38-8e006351cd70","Type":"ContainerDied","Data":"0b5bba51101271b0d75230492eb42057a4f5db9690211ef36249c40444012a3f"} Nov 03 23:30:03.620759 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.620737 5051 generic.go:356] "Generic (PLEG): container finished" podID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerID="cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8" exitCode=60 Nov 03 23:30:03.620869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.620822 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerDied","Data":"cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8"} Nov 03 23:30:03.620914 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.620875 5051 scope.go:122] "RemoveContainer" containerID="a3ea37bfb7d8cbab717c4d67e3dcad3122c5b85b7633e42e9d51bbaa22e0d37c" Nov 03 23:30:03.621195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:03.621179 5051 scope.go:122] "RemoveContainer" containerID="cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8" Nov 03 23:30:03.621464 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:30:03.621437 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-ksjgc_odh-ai-gateway-infra(a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" Nov 03 23:30:04.628122 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.628086 5051 scope.go:122] "RemoveContainer" containerID="cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8" Nov 03 23:30:04.628555 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:30:04.628360 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-ksjgc_odh-ai-gateway-infra(a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" Nov 03 23:30:04.778504 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.778479 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:04.814869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.814841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\") pod \"6718947d-9032-4f25-ab38-8e006351cd70\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " Nov 03 23:30:04.815069 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.814976 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\") pod \"6718947d-9032-4f25-ab38-8e006351cd70\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " Nov 03 23:30:04.815150 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.815076 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\" (UniqueName: \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\") pod \"6718947d-9032-4f25-ab38-8e006351cd70\" (UID: \"6718947d-9032-4f25-ab38-8e006351cd70\") " Nov 03 23:30:04.815268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.815242 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca" pod "6718947d-9032-4f25-ab38-8e006351cd70" (UID: "6718947d-9032-4f25-ab38-8e006351cd70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:04.815411 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.815385 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-openshift-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:30:04.817383 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.817354 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc" pod "6718947d-9032-4f25-ab38-8e006351cd70" (UID: "6718947d-9032-4f25-ab38-8e006351cd70"). InnerVolumeSpecName "kube-api-access-8xdnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:04.817481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.817384 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca" pod "6718947d-9032-4f25-ab38-8e006351cd70" (UID: "6718947d-9032-4f25-ab38-8e006351cd70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:04.916107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.916016 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xdnc\" (UniqueName: \"kubernetes.io/projected/6718947d-9032-4f25-ab38-8e006351cd70-kube-api-access-8xdnc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:30:04.916107 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:04.916048 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6718947d-9032-4f25-ab38-8e006351cd70-maas-api-xks-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:30:05.634829 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:05.634795 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" Nov 03 23:30:05.635286 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:05.634831 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n7ndt" event={"ID":"6718947d-9032-4f25-ab38-8e006351cd70","Type":"ContainerDied","Data":"5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f"} Nov 03 23:30:05.635286 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:05.634884 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef896d2a45effd35e94d9189641397f43f831fc6aea05394f5200a5f9d9ef3f" Nov 03 23:30:15.417074 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:15.413269 5051 scope.go:122] "RemoveContainer" containerID="cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8" Nov 03 23:30:16.714337 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:16.714304 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerDied","Data":"075ff7a402d0d6ea78faa4415694f2ca8ef092b13960e7fe7aa7b624cade8614"} Nov 03 23:30:16.714831 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:16.714379 5051 scope.go:122] "RemoveContainer" containerID="cc91c82750b13b99fb937f3a942ead85ea5898192d99f6a5f09ae898965edeb8" Nov 03 23:30:16.714831 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:16.714228 5051 generic.go:356] "Generic (PLEG): container finished" podID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerID="075ff7a402d0d6ea78faa4415694f2ca8ef092b13960e7fe7aa7b624cade8614" exitCode=60 Nov 03 23:30:16.714831 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:16.714821 5051 scope.go:122] "RemoveContainer" containerID="075ff7a402d0d6ea78faa4415694f2ca8ef092b13960e7fe7aa7b624cade8614" Nov 03 23:30:16.715118 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:30:16.715098 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895810-ksjgc_odh-ai-gateway-infra(a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" Nov 03 23:30:17.839369 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:17.839333 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:17.982127 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:17.982099 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:18.059520 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.059493 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\") pod \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " Nov 03 23:30:18.059730 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.059583 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\" (UniqueName: \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\") pod \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\" (UID: \"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea\") " Nov 03 23:30:18.059894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.059871 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca" pod "a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" (UID: "a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:18.062031 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.061978 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc" pod "a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" (UID: "a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea"). InnerVolumeSpecName "kube-api-access-8x8gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:18.160489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.160452 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x8gc\" (UniqueName: \"kubernetes.io/projected/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-kube-api-access-8x8gc\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:30:18.160489 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.160483 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea-maas-api-service-ca\") on node \"ip-10-0-135-87.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:30:18.729363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.729282 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" Nov 03 23:30:18.729363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.729308 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc" event={"ID":"a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea","Type":"ContainerDied","Data":"e440984f4d1c4d37c42193f268b960cde7ced215f566a377f028c5e6a494dda9"} Nov 03 23:30:18.729363 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.729364 5051 scope.go:122] "RemoveContainer" containerID="075ff7a402d0d6ea78faa4415694f2ca8ef092b13960e7fe7aa7b624cade8614" Nov 03 23:30:18.893192 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.893161 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:18.897587 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:18.897562 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895810-ksjgc"] Nov 03 23:30:20.406772 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:20.406740 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" path="/var/lib/kubelet/pods/a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea/volumes" Nov 03 23:30:21.621942 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:30:21.621908 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:30:24.903003 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:24.902923 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l4w64_743264a8-12f2-4ae4-9499-17f03b60978d/manager/0.log" Nov 03 23:30:27.057735 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:27.057698 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl"] Nov 03 23:30:27.107281 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:27.107248 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a59534-s8xfl"] Nov 03 23:30:28.402614 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:28.402585 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dc72k_cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd/revoke-keys/0.log" Nov 03 23:30:28.407415 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:28.407391 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41807ea-d48c-472c-92cd-6aac85d57e54" path="/var/lib/kubelet/pods/c41807ea-d48c-472c-92cd-6aac85d57e54/volumes" Nov 03 23:30:28.633209 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:28.633164 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n7ndt_6718947d-9032-4f25-ab38-8e006351cd70/revoke-keys/0.log" Nov 03 23:30:28.848742 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:28.848667 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-k7jk8_0f01b39f-3ac7-40df-a7fa-d1118c215f3d/revoke-keys/0.log" Nov 03 23:30:29.068136 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:29.068100 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r7z88_1091c07f-30cf-4e09-a7a3-06a236aa4c59/postgres/0.log" Nov 03 23:30:31.061655 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:31.061611 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t27g8_ecd47f33-4a76-447c-b457-17597360a16a/manager/0.log" Nov 03 23:30:31.872447 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:31.872416 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bxz7f_74946b97-2ba2-465d-ac92-aea0b2b87a14/manager/0.log" Nov 03 23:30:33.397442 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:33.397406 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh_d6d71f6e-e71d-4197-b99b-327870fb690c/istio-proxy/0.log" Nov 03 23:30:34.585188 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:34.585161 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4bjss_c799271b-6fb6-4e67-9e4a-45f296bf84db/istio-proxy/0.log" Nov 03 23:30:36.341640 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:36.341612 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zhbvw_23e4cfcd-d1db-4503-8a90-740070194e2b/istio-proxy/0.log" Nov 03 23:30:36.625307 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:36.625274 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zkrw7_f6b0c86f-7342-49af-a069-ed9ff722f54b/istio-proxy/0.log" Nov 03 23:30:36.849326 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:36.849294 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv_abf28985-5cc8-42ba-b374-f44d7704a0d3/istio-proxy/0.log" Nov 03 23:30:37.531398 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:37.531371 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79487cbf4c-klb6d_17f1540d-7f49-4949-9679-4c6ca2ba9de9/router/0.log" Nov 03 23:30:38.688617 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:38.688567 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-4sx8l_ce20667b-c9f6-4f80-81a0-f63d1b87d9d3/main/0.log" Nov 03 23:30:39.126897 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:39.126871 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-r7qkw_69267d17-a659-4d2d-ae3b-2560c2968dec/main/0.log" Nov 03 23:30:39.623437 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:39.623407 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2wvgk_feb10c14-99bd-468d-932f-a4cb23eec994/main/0.log" Nov 03 23:30:40.106047 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:40.106020 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv_a2e1a9bd-ec01-49a9-92ef-6688291fb567/main/0.log" Nov 03 23:30:50.810503 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:50.810464 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-l4w64_743264a8-12f2-4ae4-9499-17f03b60978d/manager/0.log" Nov 03 23:30:51.623976 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:30:51.623938 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:30:53.700731 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:53.700697 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dc72k_cc4e4b9a-7274-468b-8dd6-5ef953cb8cdd/revoke-keys/0.log" Nov 03 23:30:53.917688 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:53.917656 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n7ndt_6718947d-9032-4f25-ab38-8e006351cd70/revoke-keys/0.log" Nov 03 23:30:54.147059 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:54.147033 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-k7jk8_0f01b39f-3ac7-40df-a7fa-d1118c215f3d/revoke-keys/0.log" Nov 03 23:30:54.369208 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:54.369180 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r7z88_1091c07f-30cf-4e09-a7a3-06a236aa4c59/postgres/0.log" Nov 03 23:30:55.960650 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:55.960623 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t27g8_ecd47f33-4a76-447c-b457-17597360a16a/manager/0.log" Nov 03 23:30:56.720579 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:56.720546 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bxz7f_74946b97-2ba2-465d-ac92-aea0b2b87a14/manager/0.log" Nov 03 23:30:58.008710 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:58.008679 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh_d6d71f6e-e71d-4197-b99b-327870fb690c/istio-proxy/0.log" Nov 03 23:30:59.242647 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:30:59.242621 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4bjss_c799271b-6fb6-4e67-9e4a-45f296bf84db/istio-proxy/0.log" Nov 03 23:31:01.055538 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:01.055507 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zhbvw_23e4cfcd-d1db-4503-8a90-740070194e2b/istio-proxy/0.log" Nov 03 23:31:01.326268 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:01.326187 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-zkrw7_f6b0c86f-7342-49af-a069-ed9ff722f54b/istio-proxy/0.log" Nov 03 23:31:01.557195 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:01.557166 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-kkznv_abf28985-5cc8-42ba-b374-f44d7704a0d3/istio-proxy/0.log" Nov 03 23:31:02.198275 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:02.198249 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79487cbf4c-klb6d_17f1540d-7f49-4949-9679-4c6ca2ba9de9/router/0.log" Nov 03 23:31:03.113134 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:03.113104 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-4sx8l_ce20667b-c9f6-4f80-81a0-f63d1b87d9d3/main/0.log" Nov 03 23:31:03.608070 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:03.608043 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-r7qkw_69267d17-a659-4d2d-ae3b-2560c2968dec/main/0.log" Nov 03 23:31:04.110532 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:04.110487 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2wvgk_feb10c14-99bd-468d-932f-a4cb23eec994/main/0.log" Nov 03 23:31:04.651386 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:04.651353 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4mqfv_a2e1a9bd-ec01-49a9-92ef-6688291fb567/main/0.log" Nov 03 23:31:12.706239 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:12.706206 5051 scope.go:122] "RemoveContainer" containerID="635d655680173cbe7b9011ab3d9c4bca4c9fa6dceeed78028f34dacc3cb07b83" Nov 03 23:31:12.715191 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:31:12.715170 5051 scope.go:122] "RemoveContainer" containerID="973a9d56eb955126ea717e3212d4f39a4c33e25da16565f199be326c4184ed09" Nov 03 23:31:21.622448 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:31:21.622410 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:31:51.623103 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:31:51.623013 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:32:07.473272 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:07.473245 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:32:07.506509 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:07.506486 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-t8ztc_0cdc2bed-bef5-4115-bf4a-1abfb29cfdc3/console-operator/3.log" Nov 03 23:32:12.743380 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:12.743342 5051 scope.go:122] "RemoveContainer" containerID="04253ffeb8756e02336f1252596ac2fa97816d1a2109657fdeb174440178ee1b" Nov 03 23:32:14.365871 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.365828 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9d6v/must-gather-7twk6"] Nov 03 23:32:14.366472 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366450 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366476 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366506 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366514 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366526 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366533 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366553 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6718947d-9032-4f25-ab38-8e006351cd70" containerName="revoke-keys" Nov 03 23:32:14.366562 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366560 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6718947d-9032-4f25-ab38-8e006351cd70" containerName="revoke-keys" Nov 03 23:32:14.366939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366688 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6718947d-9032-4f25-ab38-8e006351cd70" containerName="revoke-keys" Nov 03 23:32:14.366939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366710 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366723 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.366939 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.366733 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02cc4b3-8541-46b2-80ab-2e9b5c6fb2ea" containerName="cleanup" Nov 03 23:32:14.371441 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.371420 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.401669 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.401640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19f89bd1-2bbf-4cb5-af7f-92396d6db765-must-gather-output\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.401830 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.401811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2fp\" (UniqueName: \"kubernetes.io/projected/19f89bd1-2bbf-4cb5-af7f-92396d6db765-kube-api-access-hm2fp\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.472764 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.472733 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9d6v\"/\"kube-root-ca.crt\"" Nov 03 23:32:14.472932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.472800 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x9d6v\"/\"default-dockercfg-z6rsm\"" Nov 03 23:32:14.472932 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.472809 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9d6v\"/\"openshift-service-ca.crt\"" Nov 03 23:32:14.475350 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.475332 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/must-gather-7twk6"] Nov 03 23:32:14.502471 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.502443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2fp\" (UniqueName: \"kubernetes.io/projected/19f89bd1-2bbf-4cb5-af7f-92396d6db765-kube-api-access-hm2fp\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.502619 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.502490 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19f89bd1-2bbf-4cb5-af7f-92396d6db765-must-gather-output\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.502801 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.502785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19f89bd1-2bbf-4cb5-af7f-92396d6db765-must-gather-output\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.563823 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.563799 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2fp\" (UniqueName: \"kubernetes.io/projected/19f89bd1-2bbf-4cb5-af7f-92396d6db765-kube-api-access-hm2fp\") pod \"must-gather-7twk6\" (UID: \"19f89bd1-2bbf-4cb5-af7f-92396d6db765\") " pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:14.685458 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:14.685363 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/must-gather-7twk6" Nov 03 23:32:15.060130 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:15.060100 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/must-gather-7twk6"] Nov 03 23:32:15.062981 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:32:15.062946 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f89bd1_2bbf_4cb5_af7f_92396d6db765.slice/crio-379140f4ba9ba99d00bcd4ade134534318dcd135fba893a5071b3f1fcbc15d39 WatchSource:0}: Error finding container 379140f4ba9ba99d00bcd4ade134534318dcd135fba893a5071b3f1fcbc15d39: Status 404 returned error can't find the container with id 379140f4ba9ba99d00bcd4ade134534318dcd135fba893a5071b3f1fcbc15d39 Nov 03 23:32:15.064958 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:15.064938 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:32:15.363846 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:15.363801 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/must-gather-7twk6" event={"ID":"19f89bd1-2bbf-4cb5-af7f-92396d6db765","Type":"ContainerStarted","Data":"379140f4ba9ba99d00bcd4ade134534318dcd135fba893a5071b3f1fcbc15d39"} Nov 03 23:32:16.374008 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:16.372759 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/must-gather-7twk6" event={"ID":"19f89bd1-2bbf-4cb5-af7f-92396d6db765","Type":"ContainerStarted","Data":"75578193e6a7f0eb901a196ce01ae296dc67c68aaeba479e9474135a79a47f1b"} Nov 03 23:32:17.380408 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:17.380363 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/must-gather-7twk6" event={"ID":"19f89bd1-2bbf-4cb5-af7f-92396d6db765","Type":"ContainerStarted","Data":"70372bbdbb9c289c631c2fec9bd7666b78f8b2bdf6e9c632ba86234fb714f3bf"} Nov 03 23:32:17.498222 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:17.498153 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9d6v/must-gather-7twk6" podStartSLOduration=2.33070062 podStartE2EDuration="3.498133326s" totalImagesPullingTime="1.167432706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:17.494552755 +0000 UTC m=+4211.231896985" watchObservedRunningTime="2026-11-03 23:32:17.498133326 +0000 UTC m=+4211.235477551" Nov 03 23:32:21.624267 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:32:21.624225 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:32:23.178021 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:23.176180 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dchqn_71c0cee1-1b74-43c7-8bd2-e9ec84ddb0bb/global-pull-secret-syncer/0.log" Nov 03 23:32:25.001011 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:24.998724 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9dn28_4dae9dcf-d659-41b6-8b8c-8ce5d2f5a985/connection-checker/0.log" Nov 03 23:32:25.428875 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:25.428754 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l9j49_59f23597-a6bb-4e60-a54f-b7411e073675/connection-checker/0.log" Nov 03 23:32:25.865648 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:25.865617 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-z4nhs_4657e9d9-df67-410c-8154-655ecd80a494/connection-checker/0.log" Nov 03 23:32:26.883289 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:26.883261 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mlhk9_661b34e8-d977-46ae-a3c6-6f09a1182f8d/konnectivity-agent/0.log" Nov 03 23:32:28.349035 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:28.348979 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-87.us-west-1.compute.internal_6400a77c4e4822387f7b4ea34fe62bb6/haproxy/0.log" Nov 03 23:32:33.832287 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.832248 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd"] Nov 03 23:32:33.839396 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.839368 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.866539 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.866492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-lib-modules\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.866728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.866557 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc6z\" (UniqueName: \"kubernetes.io/projected/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-kube-api-access-5sc6z\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.866728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.866640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-proc\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.866728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.866683 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-sys\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.866728 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.866707 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-podres\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.925869 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.925819 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd"] Nov 03 23:32:33.967667 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967636 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-lib-modules\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967667 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967675 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc6z\" (UniqueName: \"kubernetes.io/projected/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-kube-api-access-5sc6z\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967699 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-proc\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-sys\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-podres\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-lib-modules\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967844 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-sys\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-proc\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:33.967934 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:33.967848 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-podres\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:34.028894 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:34.028855 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc6z\" (UniqueName: \"kubernetes.io/projected/e281a69e-27b8-4fde-8c5a-679bc4d2b6db-kube-api-access-5sc6z\") pod \"perf-node-gather-daemonset-crvdd\" (UID: \"e281a69e-27b8-4fde-8c5a-679bc4d2b6db\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:34.038087 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:34.038061 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t27g8_ecd47f33-4a76-447c-b457-17597360a16a/manager/0.log" Nov 03 23:32:34.153123 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:34.153090 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:34.531491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:34.531464 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd"] Nov 03 23:32:34.533397 ip-10-0-135-87 kubenswrapper[5051]: W1103 23:32:34.533370 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode281a69e_27b8_4fde_8c5a_679bc4d2b6db.slice/crio-daaff8296799d3df096ff9fdea4be734fc2209ce1e348b61779825b9dffc7626 WatchSource:0}: Error finding container daaff8296799d3df096ff9fdea4be734fc2209ce1e348b61779825b9dffc7626: Status 404 returned error can't find the container with id daaff8296799d3df096ff9fdea4be734fc2209ce1e348b61779825b9dffc7626 Nov 03 23:32:35.330550 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:35.330523 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bxz7f_74946b97-2ba2-465d-ac92-aea0b2b87a14/manager/0.log" Nov 03 23:32:35.542296 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:35.542258 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" event={"ID":"e281a69e-27b8-4fde-8c5a-679bc4d2b6db","Type":"ContainerStarted","Data":"1d9767c97d3f0604e637eb84902fdc6f36fe94136d1da5769f425e81c179fae2"} Nov 03 23:32:35.542481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:35.542310 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" event={"ID":"e281a69e-27b8-4fde-8c5a-679bc4d2b6db","Type":"ContainerStarted","Data":"daaff8296799d3df096ff9fdea4be734fc2209ce1e348b61779825b9dffc7626"} Nov 03 23:32:35.542481 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:35.542471 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:35.659073 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:35.659020 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" podStartSLOduration=2.659006863 podStartE2EDuration="2.659006863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:35.657539659 +0000 UTC m=+4229.394883887" watchObservedRunningTime="2026-11-03 23:32:35.659006863 +0000 UTC m=+4229.396351079" Nov 03 23:32:41.560695 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:41.560667 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-crvdd" Nov 03 23:32:41.720403 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:41.720368 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6jdkv_95598dfe-8d0c-4a37-9a8f-479bb69f07c0/cluster-monitoring-operator/0.log" Nov 03 23:32:45.507483 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:45.507456 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqdzw_0895fe48-a6af-47c0-894c-8d0940027e95/node-exporter/0.log" Nov 03 23:32:45.921163 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:45.921131 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqdzw_0895fe48-a6af-47c0-894c-8d0940027e95/kube-rbac-proxy/0.log" Nov 03 23:32:46.334896 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:46.334817 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqdzw_0895fe48-a6af-47c0-894c-8d0940027e95/init-textfile/0.log" Nov 03 23:32:51.624043 ip-10-0-135-87 kubenswrapper[5051]: E1103 23:32:51.624002 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-88khg" containerName="extractor" Nov 03 23:32:53.004747 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:53.004720 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-zxg64_332253ec-6bf3-4703-9cb0-f87db7afa536/prometheus-operator-admission-webhook/0.log" Nov 03 23:32:56.486491 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:56.486459 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f7xkbnh_d6d71f6e-e71d-4197-b99b-327870fb690c/istio-proxy/0.log" Nov 03 23:32:57.957903 ip-10-0-135-87 kubenswrapper[5051]: I1103 23:32:57.957870 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4bjss_c799271b-6fb6-4e67-9e4a-45f296bf84db/istio-proxy/0.log"